We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Exploring trace data

Trace datasets are exactly the same as any other dataset in Honeycomb: you can break down, filter and aggregate by any field.

To view traces associated with your query, select the “Traces” tab. This will show you the slowest traces that match the query, calculated by root span duration.

trace summary table

The “Traces” tab will show a maximum of 10 sample traces. If you need to access a larger list of traces that match your query, you can click the “Raw Data” tab and click any trace ID to view the trace that includes that event.

raw data mode table with clickable trace id

Alternatively, add a Break Down by traceId to your query. (Use trace.trace_id if you’re using a Beeline.) Click on a traceId in the results table to see its trace diagram.

If you want to see specific traces that correspond to a feature on the graph, such as a latency spike, you can select a region on the graph, and the list of trace summaries will update:

zoomed graph selection

Click on one of the traceId links to see the full waterfall diagram for that trace.

trace ID link

The first row is the root span, the parent of the entire trace. Below it are each of the spans in this trace. The bars on the right display how much time each operation took to execute. This type of graph is also called a waterfall diagram, as it shows how the order of operations cascade across the total execution time.

trace diagram

To see more detail about an individual span, click on a row to view that span’s raw data in the right sidebar:

trace sidebar

To display additional columns, open the column selection menu by clicking the plus icon in the header. Check the boxes for the columns you’d like to include. To hide a column, click on the column name and select “Hide column” from the menu. To collapse or expand a portion of the trace, click on the arrow on the left side of a span row.

Example: slow API calls

Here is an example of investigating a problem using tracing:

A few users are reporting slow API calls. The initial investigation shows one endpoint is periodically slow. One user is making a lot of calls, and that is slowing down responses for everyone.

query results showing one user making lots of requests to /api/v2/tickets/export

Selecting a representative trace shows the full execution of the API request, helping us understand why these API requests are slow.

example slow trace

The first part seems normal, but ticketBackend is making more calls than expected to mysql. Clicking on one of the mysql spans shows more information about it in the sidebar, including the specific database query.

trace showing ticketBackend calling mysql many times sequentially

The information from this trace points to a possible next step. Is the user running a script gone wrong? Or maybe this endpoint needs tighter rate limiting. Traces show a level of detail that can be hard to see otherwise.