Traces describe a complete unit of work for one or more services in an Environment. Each trace is composed of multiple spans. A span is a unit of instrumentation from a single location in your code. A trace allows you to inspect the contextual data captured by each of its spans to discover how your systems performed in a given context.
Traces enable you to:
The trace detail view is accessible through points in a visualization, the trace button , or directly with a permalink.
In Query Builder, VISUALIZE with any clause. This creates the visualization below the Query Builder.
Select a point in the visualization. Choose from a line graph or from a Heatmap.
From a line graph:
From a Heatmap:
In the menu that appears, select View trace.
The next screen displays a trace detail view.
The displayed trace is the slowest trace with a child span and with a value is less than or equal to the value of your selected point.
Therefore, selecting a point on the
p90 graph gives a different trace than selecting a point on an overlaid
You will be automatically taken to the first span in the trace waterfall that matches the query’s filter criteria. Selecting on a point in a graph with an empty WHERE clause will direct you to the root span in the trace waterfall.
Run a query in Query Builder.
Select the Traces tab that appears below any visualizations. The Traces tab displays up to 10 traces with the slowest spans that match your query’s filters.
To be sent to a trace’s trace detail view, select the trace’s trace button or its hyperlinked Trace ID.
Run a query in Query Builder.
Select the Events tab that appears below any visualizations. The Events tab displays raw data for all spans that match your query’s filters.
To be sent to a span’s trace detail view, select the span’s trace button. In the next screen, the selected span appears highlighted in blue within the trace waterfall display.
The trace detail view displays information in four areas:
Each trace has a unique identifier presented at the top of page along side navigation elements.
The left-arrow navigates away from the trace view back to the visualization or results table where the trace was selected.
Honeycomb presents a trace when it has received the root span. Each span has a unique identifier and datetime stamp of when it was created. The Reload Trace button reloads the trace to ensure the waterfall representation contains all spans in the trace.
Traces may display a warning that it is missing spans with a link to resources to assist with troubleshooting the missing spans.
The trace summary displays important metadata and provides a condensed view of the trace waterfall diagram. Metadata about the trace includes its total number of spans, the timestamp of the root span, and the total trace duration. Use this view to find long-running spans and spans with errors within your trace without scrolling through the entire trace waterfall. Expand or collapse the summary view by selecting the directional caret.
The trace summary displays up to 6 levels of span dependency. The first level in the topmost row starts with the root span. The second row shows the root span’s dependent spans, and each following row displays a dependent span level, if applicable. The width of the summary represents the whole duration of the trace.
Hover over any span to see the name of the longest running span at that depth and point in time. Select the span to view its highlighted location in the waterfall representation below.
Use Highlight errors to toggle span error highlighting on and off. Spans with errors appear in red. Select a span with an error to view its location in the waterfall representation below and additional metadata in the trace sidebar.
Honeycomb uses the metadata from each span to reconstruct the relationships between them and generate a trace diagram. This is also called a waterfall diagram because it shows how the order of operations cascades across the total execution time.
Each span contains a collection of fields and values.
The diagram displays, at a minimum, the span’s relationship, its name, and a representation of its duration.
Next to each span’s name shows its relationship in the trace. A span with dependencies displays a box with the number of dependent spans. A span without a box represents a span with no dependencies.
The currently selected span’s row is highlighted in blue. Details about the selected span appear to the right in the trace sidebar.
When a trace contains many spans, it can be useful to collapse spans that are not of interest, or expand specific parent spans to investigate. Navigate the diagram with either mouse or keyboard navigation.
The search field provides the ability to type in a field or value and highlights spans that contain your search term. Error Highlighting shows how many spans with errors exists in the trace. The Fields button customizes the fields in the waterfall representation.
To search spans, use the search box to type in the field or value you want to find. As you type, the display changes and highlights any spans that contain your search term.
A count of matching spans found appears next to the search box and navigation arrows. Use the navigation arrows to move between the spans that match the search term.
The count of spans containing errors is displayed above the trace waterfall. Select this count of spans to navigate between the spans with errors. By default, spans that contain errors are displayed in red. The error field is also highlighted at the top of the field list in the trace sidebar. Span error detection is based on the Error field as configured in the Dataset Definition.
To add a field to the waterfall representation, use the Fields button in the upper right corner and type in the target field name.
A list of suggested fields appears based on your input.
To update the waterfall representation, complete typing the field name and press
Enter, or select the target field name from the list of suggested fields.
The waterfall representation updates with a new column labeled with the field name and its information.
To remove fields in the span display, use the Fields button to display the current fields. Use the x to right of each field name to remove it. The display updates after any change.
To resize columns in the waterfall representation, drag the gray borders that separate column names to your desired width. A set minimum width exists for each column. To remove a column completely, use the Fields button.
The color for each span applies to its entire row, such as the count box, relationship tracing, and duration representation.
By default, the color is determined by
service.name if there is more than one service name defined; otherwise, they are colored by
A half-filled water drop icon appears next to the column header of the determining field.
Each distinct value in the determining field gets its own color.
Changing the determining field colors the waterfall display differently and may emphasize different attributes.
To change the determining field, select the downward arrow next to the target field’s column header. In the menu that appears, select Color rows based on the values in this field. The display refreshes with the new color scheme, and the half-filled water drop icon appears next to the selected field’s column header.
When a span is selected, the trace sidebar updates with details about the span, including its fields, span events, and links.
The Span Events tab of the trace sidebar displays details about span events, if applicable. To access, navigate to the Span Events tab and select the Span Event name to show its details. Alternatively, in the waterfall diagram, select the circle that represents a span event to display the span event’s details in the trace sidebar’s Span Events tab.
The Links tab of the trace sidebar displays details about span links, if applicable. To access, navigate to the Links tab to view span link details. Alternatively, in the waterfall diagram, select the link icon that represents a span link to display details in the trace sidebar’s Links tab.
The minigraph, at the top of the trace sidebar, shows a heatmap view of the selected span relative to others with the same fields displayed in the waterfall. Selecting anywhere in the minigraph sends you to the Query Builder display with a query that corresponds to the minigraph.
The trace sidebar includes a filter. Enter terms into this sidebar filter and the sidebar display refreshes to show matching fields and/or values. This filter remains in the trace sidebar as you select different spans. Use this behavior to track a particular field as you move between a set of spans.
Any span’s field or value can act as an entry point back to the Query Builder. In the Trace Sidebar, select the three dots to the right of a targeted field to open its context menu. In addition to copy options, the menu includes GROUP BY and WHERE actions, which create a new query that re-uses the original query but adds the selected clause and field.
The new query may not return results.
If the fields you select are only found on child spans, but the original query includes
trace.parent_id does-not-exist or
is_root, then the new query will be self-contradictory, and will not return any results.
Remove any unneeded clauses to get a full result.
When collapsing the tree, it can be difficult to see the extent of the child spans of a parent.
For example, an async process might have a parent that returns in a few milliseconds – but it starts a database query that runs for several minutes.
Honeycomb draws a thin line to the right of the parent span to show the duration of the child process.
For example, in this figure, the span with
head.startQuery runs 2.533 ms – but it has children that run much longer.
(The three subprocesses called
FetchPartial all run over 2.5 seconds).
Clock skew, temporal anomalies, or time travel can cause a child to start before the parent span. Honeycomb renders these with similar lines to the left of the main body. To learn more about how to resolve this issue, visit our Troubleshooting Traces documentation.
Sometimes, a trace has some spans that claim to have a
trace.parent_id – but no span claims to have that
Honeycomb renders a missing span in that slot: a placeholder that shows the span with all the children that point to that parent.
Honeycomb also expects one span to have no parent; that is the root span.
If there is no root span, then this is understood as a missing root; in that case, the root span is drawn as missing.
To learn more about how to resolve this issue, visit our Troubleshooting Traces documentation.
Honeycomb displays traces with a maximum of 32,000 spans. If you load a trace larger than 32,000 spans into the trace viewer, Honeycomb will display the 32,000 spans closest to the root, using a breadth-first search, and will display a warning message. Honeycomb will omit the remainder of the spans. In addition, Honeycomb will display placeholder spans showing where the omitted spans would go.
To work with very large traces, use the
CONCURRENCY aggregate to visualize the structure of the trace, as seen in the Trace Summary example.
Here is an example of investigating a problem using tracing:
A few users are reporting slow API calls. The initial investigation shows one endpoint is periodically slow. One user is making a lot of calls, and that is slowing down responses for everyone.
Selecting a representative trace shows the full execution of the API request, helping us understand why these API requests are slow.
The first part seems normal, but
ticketBackend is making more calls than expected to
Clicking on one of the
mysql spans shows more information about it in the sidebar, including the specific database query.
The information from this trace points to a possible next step. Is the user running a script gone wrong? Or maybe this endpoint needs tighter rate limiting. Traces show a level of detail that can be hard to see otherwise.