Traces with OpenTelemetry
Send traces from Go, Python, Rust, Java or Node with their own OpenTelemetry SDK, and see slow routes, slow queries, N+1 queries and regressions by release.
Send traces from any backend with its own OpenTelemetry SDK: Go, Python, Rust, Java, Node or anything else that speaks OTLP over HTTP. The Performance page then shows your slowest routes and query shapes, each request's waterfall, N+1 queries, and the routes that got slower in a release.
Endpoint and key
- Endpoint:
https://in.littlefriend.io/v1/traces, OTLP/HTTP in protobuf (the default of most SDKs) or JSON, gzipped or not. - Header:
Authorization: Bearer lfs_..., your project's secret server key. Create one under Settings, Keys, or withlittlefriend keys create --kind server. Keep it on the server. - Service and release: the resource attribute
service.namenames the service, andservice.versionis its release, which regressions compare.deployment.environment.nameis kept as the environment.
Every SDK reads the standard environment variables, so often no code changes. In the headers variable the space after Bearer is written %20.
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://in.littlefriend.io/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20lfs_YOUR_SERVER_KEY"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME=api
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment.name=production"Go
// go get go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
exporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpointURL("https://in.littlefriend.io/v1/traces"),
otlptracehttp.WithHeaders(map[string]string{
"Authorization": "Bearer " + os.Getenv("LF_SERVER_KEY"),
}),
)
if err != nil {
log.Fatal(err)
}
res, _ := resource.Merge(resource.Default(), resource.NewWithAttributes(semconv.SchemaURL,
semconv.ServiceName("api"),
semconv.ServiceVersion("1.4.2"),
))
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter), sdktrace.WithResource(res))
defer tp.Shutdown(ctx)
otel.SetTracerProvider(tp)Python
# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
exporter = OTLPSpanExporter(
endpoint="https://in.littlefriend.io/v1/traces",
headers={"Authorization": f"Bearer {os.environ['LF_SERVER_KEY']}"},
)
provider = TracerProvider(resource=Resource.create({"service.name": "api", "service.version": "1.4.2"}))
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)Rust
// Cargo.toml: opentelemetry, opentelemetry_sdk, and opentelemetry-otlp with features ["http-proto", "reqwest-client"]
use opentelemetry_otlp::{SpanExporter, WithExportConfig, WithHttpConfig};
use opentelemetry_sdk::{trace::SdkTracerProvider, Resource};
use std::collections::HashMap;
let exporter = SpanExporter::builder()
.with_http()
.with_endpoint("https://in.littlefriend.io/v1/traces")
.with_headers(HashMap::from([(
"Authorization".to_string(),
format!("Bearer {}", std::env::var("LF_SERVER_KEY")?),
)]))
.build()?;
let provider = SdkTracerProvider::builder()
.with_batch_exporter(exporter)
.with_resource(Resource::builder().with_service_name("api").build())
.build();
opentelemetry::global::set_tracer_provider(provider);Java
// io.opentelemetry:opentelemetry-sdk and io.opentelemetry:opentelemetry-exporter-otlp
OtlpHttpSpanExporter exporter = OtlpHttpSpanExporter.builder()
.setEndpoint("https://in.littlefriend.io/v1/traces")
.addHeader("Authorization", "Bearer " + System.getenv("LF_SERVER_KEY"))
.build();
SdkTracerProvider provider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.setResource(Resource.getDefault().merge(Resource.create(Attributes.of(
AttributeKey.stringKey("service.name"), "api",
AttributeKey.stringKey("service.version"), "1.4.2"))))
.build();
OpenTelemetrySdk.builder().setTracerProvider(provider).buildAndRegisterGlobal();Or with the OpenTelemetry Java agent and no code at all:
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://in.littlefriend.io/v1/traces \
OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer%20$LF_SERVER_KEY" \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_SERVICE_NAME=api \
java -javaagent:opentelemetry-javaagent.jar -jar app.jarNode
// npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto @opentelemetry/auto-instrumentations-node
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { NodeSDK } from '@opentelemetry/sdk-node';
const sdk = new NodeSDK({
serviceName: 'api',
traceExporter: new OTLPTraceExporter({
url: 'https://in.littlefriend.io/v1/traces',
headers: { Authorization: `Bearer ${process.env.LF_SERVER_KEY}` },
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();With @littlefriend/node you can skip this: it sends the same spans with the same server key.
What is kept
A span keeps its trace and span ids, its parent, its timing, its kind, its status and these attributes. Every other attribute is dropped when it arrives, and so are span events, links, status messages, headers, cookies and form values.
| Attribute | Also read from |
|---|---|
db.system | db.system.name |
db.query.text | db.statement |
db.operation.name | db.operation |
db.collection.name | db.sql.table, db.mongodb.collection, db.cosmosdb.container |
db.response.returned_rows | |
http.route | |
http.request.method | http.method |
http.response.status_code | http.status_code |
url.full | http.url |
server.address | net.peer.name, http.host |
turbine.model | |
turbine.operation | |
error.type | exception.type |
- Statements are kept only as their shape: every value is replaced by
?and lists collapse, soWHERE id IN (1, 2, 3)is stored asWHERE id IN (?). This happens again at the collector whatever your SDK sent. - URLs lose their query and fragment, and ids in the path read
:id. - Span names are rebuilt from the route, the query or the host where the span has one, and otherwise cleaned the same way.
A request may carry up to 4 MB and 10,000 spans. Spans with bad ids or times are refused with an OTLP partial success, which exporters do not retry.
Sampling and the daily cap
A project keeps 100% of traces by default, up to 1,000,000 spans per UTC day. Lower the share or change the cap (1,000 to 20,000,000) on the Performance page, or with the settings API. Sampling is decided by trace id, so every service keeps or drops the same traces and a kept trace is whole. Past the cap, the rest of the day's spans are refused with a partial success until midnight UTC.
curl -X PUT https://api.littlefriend.io/api/projects/prj_YOUR_PROJECT/performance/settings \
-H "Authorization: Bearer $LITTLEFRIEND_TOKEN" -H "Content-Type: application/json" \
-d '{"samplePercent": 25, "dailySpanCap": 2000000}'Retention
Raw spans, the waterfalls and N+1 findings are kept 7 days. Hourly latency histograms per route and per query shape are kept as long as your other totals, so p50, p95 and p99 over any range are exact to within one histogram step.
Read it
- N+1: one query shape run 5 or more times within one trace, listed per route with the trace that repeated it most.
- Regressions: a route or query shape whose p95 is at least 20% and 20 ms slower than in the release before, with at least 20 spans on each side.
- One trace: the dashboard opens it at
/p/<project>/performance/traces/<trace id>.
From the terminal, or ask an AI assistant connected over MCP:
littlefriend perf routes --sort total
littlefriend perf queries --limit 10
littlefriend perf n-plus-one --route /orders/:id
littlefriend perf releases
littlefriend perf trace 4bf92f3577b34da6a3ce929d0e0e4736