We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Troubleshooting your trace data

Common questions

Traces appear incomplete or aren’t showing up in UI

If you’re not seeing the complete trace (you seem to be missing spans within the trace), it could be one of several items.

Verify the timestamps on each span

You must set the timestamp to the start of the span and provide the duration of the span as a part of the span record. Within your system, double check that the timestamp is being set correctly (e.g. not set to 1970-01-01T00:00:00 or some other unexpected value).

Ensure the query time window is large enough

Once you have verified the timestamps are set correctly, be aware that we currently only locate spans in the time range of the original query. So if, for example, you do a query of “Last 30 minutes” and then click a traceId, we only locate the spans in that 30 minute window. If some of the span timestamps are older than that, you won’t see them. Ensure the query time window is large enough to show all the spans within your trace.

Check to see if the OpenTracing proxy is dropping events

If you are using the honeycomb-opentracing-proxy to send traces to Honeycomb, that proxy is designed to drop events under pressure rather than fall behind (most opentracing client libraries behave similarly). The proxy will log to stdout if it does drop anything; you’ll see a message “Error sending span to Honeycomb” with more details if that has happened.

Note that events sent to Honeycomb will almost always be queryable in the Honeycomb UI within a few seconds after you send them. Feel free to reach out to us if you’d like us to check if we’re receiving the events. We’ll be able to tell you if the event is even making it to our collectors.

Honeycomb isn’t recognizing my traces

Note: depending on your configuration, field names may be slightly different. In addition, you can manually select columns as tracing fields, as explained in changing the trace schema.

Beeline Tracing Manual Tracing Default
trace.span_id id
trace.trace_id traceId
trace.parent_id parentId
duration_ms durationMS
service_name serviceName
name name

When Honeycomb isn’t recognizing your traces, it’s typically caused by a few common pitfalls.

First, to verify that Honeycomb is correctly detecting your trace, check that the traceId field is hyperlinked. To do this, select traceId in the GROUP BY window and COUNT in the VISUALIZE window.

If the traceId field looks like it does below - as a quoted string - Honeycomb is not properly recognizing your traces.

Troubleshooting Tips

Are you properly using the column “parentId”?

Honeycomb recognizes a dataset as a tracing dataset by the existence of the column “parentId”. You must send at least one event that has the column “parentId” in it or Honeycomb will not recognize the dataset as a tracing dataset and will not show traces.

But, remember that root spans should not have the column “parentId” in order to be recognized as root spans.

This likely raises a question for you: what if I’m sending a traces that only have a single span? As root spans, they’re not allowed to have a “parentId” column if they’re to be detected properly, so how will my dataset get a parentId column?

If this is your situation (and it very well may be if you’re experimenting with tracing), just send in any event with a column parentId. All you need is one event to be sent into Honeycomb with parentId as a field and the column will be created. Then, Honeycomb will properly recognize your dataset as a tracing dataset.

Alternately, by manually changing the trace schema, you can ensure that the trace schema is recognized.

Are you using the right field names?

Honeycomb automatically supports two different tracing field name sets (known as schemas) - one for auto-magic tracing done by the Beelines and one for manually configured tracing. These different schemas cannot be mixed. If you are using the manual tracing schema, then the presence of any field names from the Beeline schema will cause Honeycomb to not recognize your traces.

Some users, while experimenting with tracing, inadvertently include a mix of field names from the two schemas. Double check that you do not have this problem.

For example, let’s say you have the following columns in your dataset: “id”, “traceId”, “parentId”, “durationMs”, and “trace.span_id”. This is mixing the beeline schema with the manual schema. Traces will not show up correctly.

If you do have a mix of schemas, you can correct this issue by manually changing the trace schema on your team’s dataset settings page under the tracing tab.

Alternately, you may hide fields until there is only one valid tracing schema. To fix this, simply go into your Dataset Schema page (shown below) and click Hide on the erroneous field - in our example, “trace.span_id”.

Verify field formats.

Every trace event must have a defined trace identifier (traceId), a span identifier (spanId), and a duration (durationMs).

The traceId and spanId must both be strings; the duration must be a number, measuring milliseconds.

Case matters

Make sure the case matches exactly what is in the documentation. Specifically notice the capitalization of Id and id.

Spans in a trace appear out of order or have strange time offsets

The tracing view appears to show out of order spans, there are gaps in the layout of the spans or the root span is missing.

There are a number of reasons why spans in a trace may appear out of order or have strange time offsets.

Make sure you’re providing the start time of the span

When sending trace events into Honeycomb, you must include a “timestamp” field in the event that represents the start time of the span.

If you do not do this, Honeycomb will set the timestamp for the event to when that event was received by our collectors. This could cause the spans in a trace to appear out of order or have strange time offsets.

Verify that the parentId, traceId and id are set properly

The root span for any given trace must have no field for “parentId” in its event. If all of the spans in a trace have a “parentId”, Honeycomb will not show a root span for that trace.

Honeycomb identifies which span is the root span by its absence of the column “parentId”. If all of your span events have a “parentId” column, then Honeycomb assumes the root span is missing.

Thus, first, ensure that your root spans have no “parentId” field and validate that your child spans have their “parentId” and “id” set properly as well.

Verify that duration is set properly

You must include the duration of the span, in milliseconds (in the field durationMs). Verify that this is being set properly.