Back in 2020, when we laid out where Netdata was going, the plan listed anomaly detection, log collection and “application traces collection”. The first two shipped years ago. Traces took longer, because we wanted them to work the way the rest of Netdata works: collected and stored on your own nodes, with no separate backend to stand up.
Distributed tracing is now available in Netdata. The same Agent that receives your OpenTelemetry metrics and logs now receives, stores and indexes OpenTelemetry traces, and the new Traces tab in Netdata Cloud lets you go from a latency distribution to a single slow span in a few clicks.
How it works
Your applications are instrumented with an OpenTelemetry SDK, or they already send spans to an OpenTelemetry Collector. Either way, you point the exporter at a Netdata Agent over OTLP/gRPC, on the same endpoint (port 4317) that already accepts OTLP metrics and logs.
The Agent writes every span it accepts to its own disk and indexes it: span names, kinds and statuses, plus span, resource, scope, event and link attributes, and a trace-ID index so a whole trace can be assembled when you open it. Netdata doesn’t sample. If you want sampling, you configure it in your SDK or Collector, and whatever arrives is kept.
Trace data is not stored in Netdata Cloud. Cloud brokers access to the Agent that holds the spans, exactly as it does for metrics and logs, so traces stay on your infrastructure along with everything else Netdata collects. Traces have their own retention settings (size, age and number of files), separate from logs and metrics, and you can offload older trace files to S3-compatible object storage. The OTLP endpoint supports TLS and mutual TLS, and tenant selection keeps sender groups isolated with separate retention.
A walk through the Traces tab
The Traces tab sits next to Metrics and Logs in Netdata Cloud, for the same nodes. It opens on a duration heatmap: time runs left to right, trace duration is bucketed on a logarithmic scale from under a millisecond to over ten seconds, and the color intensity is the number of traces in each bucket. Error spans show up as red cells, so a burst of failures or a band of unusually slow requests stands out before you read a single number.
Hovering a cell shows how many traces and error spans it holds. Clicking it narrows the trace list below to that time slice and duration band. In the screenshot below, a single 15-second window in the 10 to 100 ms band holds 324 traces, and the list now shows only those.
The heatmap is one of several chart views. The same menu switches to trace volume, error spans, duration percentiles (p50, p95 and p99) or duration over time, depending on whether you are chasing a traffic change, an error spike or a slow tail.
The filter panel narrows by service and span name, and a minimum duration field hides anything faster than the threshold you care about. Filtering to the checkout service, for example, leaves only the traces that touched it. Each entry in the list shows the root service and operation, the span count, the services involved and the total duration, so the 96-span checkout that took 118 ms is easy to tell apart from a two-span health check.
Opening a trace gives you the full waterfall: every span across every service, nested by parent, with its timing on a shared axis, its span kind and errors highlighted. This checkout trace from the OpenTelemetry demo has 138 spans across 12 services. You can see the load generator’s request pass through the frontend proxy, the frontend, and the product catalog service down to a 752 µs database call, then the cart calls that follow it.
Clicking any span opens its details: status, start time, duration, parent span, and every attribute your instrumentation recorded, with separate Resource, Scope and Raw views (plus Events and Links when the span has them). Every trace also has a shareable link, so the trace you found is the one your teammate opens.
Getting started
If your services are already instrumented with OpenTelemetry, there is nothing new to install. Netdata’s OpenTelemetry plugin listens on 127.0.0.1:4317 by default. Most SDKs only need three environment variables:
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
The last line matters. Netdata accepts OTLP over gRPC, and many SDKs default to OTLP/HTTP. If you have services that can only export OTLP/HTTP, put an OpenTelemetry Collector in front of Netdata: it receives OTLP/HTTP from those services and forwards OTLP/gRPC to the Agent. Native OTLP/HTTP ingestion is on our roadmap.
If you already run a Collector, add a traces pipeline that exports to the Agent:
receivers:
otlp:
protocols:
grpc:
http:
exporters:
otlp_grpc/netdata:
endpoint: "127.0.0.1:4317"
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp_grpc/netdata]
The insecure setting is only for a Collector on the same host as the Agent. To accept spans from other machines, bind the endpoint beyond loopback with TLS or mutual TLS. Traces are queried on the Agent that received them, so to keep a group of services queryable together, point their SDKs, or a Collector gateway, at one Agent or Parent. The OTLP ingestion documentation covers the endpoint options, retention settings and troubleshooting.
Traces for your AI assistant
Netdata AI and Netdata’s MCP server can query traces alongside metrics, logs and alerts. An assistant investigating a latency alert can look at the slow traces from the same window instead of stopping at the CPU chart and asking you what the application was doing.
What it doesn’t do yet
Netdata’s tracing is distributed tracing on OpenTelemetry, built on the same Agent as your metrics and logs. It doesn’t draw service maps or dependency graphs, it doesn’t do code-level profiling, and it doesn’t ingest the native Jaeger or Zipkin protocols, so spans need to arrive through OpenTelemetry. Traces are explored in Netdata Cloud, so you’ll need to sign in to see them. If your team depends on a mature APM suite’s service maps or profiler, you’ll still want that for those workflows.
See it live
We demoed the Traces tab end to end in the Introducing Distributed Tracing in Netdata webinar, and the recording is available now. For a feature overview, see the distributed tracing page.
Distributed tracing is available now. Send spans to a Netdata Agent and open the Traces tab in Netdata Cloud.