Troubleshooting your trace data | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Troubleshooting your trace data

Common questions when troubleshooting your trace data:

Traces Appear Incomplete or Are Not Showing Up in the UI 

If you are not seeing the complete trace, where you seem to be missing spans within the trace, it could be one of several reasons.

Verify the Timestamps on Each Span 

You must set the timestamp to the start of the span and provide the duration of the span as a part of the span record. Within your system, double check that the timestamp is being set correctly. For example, not set to 1970-01-01T00:00:00 or some other unexpected value.

Ensure the Query Time Window is Large Enough 

Once you have verified the timestamps are set correctly, be aware that we currently only locate spans in the time range of the original query. So if, for example, you do a query of “Last 30 minutes” and then select a trace.trace_id, we only locate the spans in that 30 minute window. If some of the span timestamps are older than that, you will not see them. Ensure the query time window is large enough to show all the spans within your trace.

Check to See if an Intermediate Piece Such as the OpenTelemetry Collector is Having Issues 

Honeycomb can accept data in either Honeycomb’s JSON format over HTTP, or as OTLP format over gRPC/HTTP. Make sure that your Exporter is configured to export trace data to Honeycomb. Try running an OpenTelemetry Collector in Docker to confirm that data is being exported correctly.

Honeycomb is Not Recognizing My Traces 

Depending on your configuration, field names may be slightly different. In addition, you can manually select columns as tracing fields, as explained in changing the trace schema.

Standard Tracing Fields Legacy Zipkin Fields
trace.span_id id
trace.trace_id traceId
trace.parent_id parentId
duration_ms durationMs serviceName
name name

When Honeycomb is not recognizing your traces, it is typically caused by a few common pitfalls.

First, to verify that Honeycomb is correctly detecting your trace, check that the trace.trace_id field is hyperlinked. To do this, select trace.trace_id in the GROUP BY window and COUNT in the VISUALIZE window.

If the trace.trace_id field looks like it does below - as a quoted string - Honeycomb is not properly recognizing your traces.

Screenshot of the trace view where the trace.trace_id field appears as a quoted string

Properly Use the Column trace.parent_id 

Honeycomb recognizes a dataset as a tracing dataset by the existence of the column trace.parent_id. You must send at least one event that has the column trace.parent_id in it or Honeycomb will not recognize the dataset as a tracing dataset and will not show traces.

But, remember that root spans should not have the column trace.parent_id in order to be recognized as root spans.

This likely raises a question for you: what if I am sending a traces that only have a single span? As root spans, they are not allowed to have a trace.parent_id column if they are to be detected properly, so how will my dataset get a trace.parent_id column?

If this is your situation (and it very well may be if you are experimenting with tracing), just send in any event with a column trace.parent_id. All you need is one event to be sent into Honeycomb with trace.parent_id as a field and the column will be created. Then, Honeycomb will properly recognize your dataset as a tracing dataset.

Alternately, by manually changing the trace schema, you can ensure that the trace schema is recognized.

Verify Field Formats 

Every trace event must have a defined trace identifier (trace.trace_id), a span identifier (trace.span_id), and a duration (duration_ms).

The trace.trace_id and trace.span_id must both be strings; the duration must be a number, measuring milliseconds.

Spans in a Trace Appear Out of Order or Have Strange Time Offsets 

If the tracing view appears to show out of order spans, then there are gaps in the layout of the spans or the root span is missing.

There are a number of reasons why spans in a trace may appear out of order or have strange time offsets.

Make Sure You Are Providing the Start Time of the Span 

When sending trace events into Honeycomb, you must include a “timestamp” field in the event that represents the start time of the span.

If you do not do this, Honeycomb will set the timestamp for the event to when that event was received by our collectors. This could cause the spans in a trace to appear out of order or have strange time offsets.

Verify That trace.parent_id, trace.trace_id, and trace.span_id Are Set Properly 

The root span for any given trace must have no field for trace.parent_id in its event. If all of the spans in a trace have a trace.parent_id, Honeycomb will not show a root span for that trace.

Honeycomb identifies which span is the root span by its absence of the column trace.parent_id. If all of your span events have a trace.parent_id column, then Honeycomb assumes the root span is missing.

Thus, first, ensure that your root spans have no trace.parent_id field and validate that your child spans have their trace.parent_id and “id” set properly as well.

Verify that Duration is Set Properly 

You must include the duration of the span, in milliseconds and in the field durationMs. Verify that this duration is being set properly.

Honeycomb is not Recognizing my Services 

If you can not find your service in Honeycomb, there are a number of areas to check.

Verify that Your API key has the Write Permissions 

In order to create Service Datasets, your API key must have sent events and create dataset permissions. Learn about API Key permissions.

Verify that Your Team has not hit the Service Dataset Maximum 

Every environment has a service dataset limit. If you have reached this limit, you should check for any inaccurate Service Datasets or contact Support via email at if you need more.

Verify that Your Event has Defined a Service 

Depending on your instrumentation approach, events with undefined services receive a default value. In OpenTelemetry and Beelines, it may start with unknown_service. You can query this dataset in Honeycomb to see if your missing events are in here.

Verify OTLP Protobuf Definitions 

Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP protobuf definitions are 0.7.0 for traces and metrics.

If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.

Did you find what you were looking for?