SELECT, WHERE, and GROUP BY clauses, located at the top of the Query Builder.
SELECT- Performs a calculation and displays a corresponding graph over time. MostSELECTqueries return a line graph while theHEATMAPvisualization shows the distribution of data over timeWHERE- Filters based on attribute parameter(s)GROUP BY- Groups fields by attribute parameter(s)
Number of Total Root Spans
This query calculates the total number of root spans received by your application by looking for all spans without a parent span ID.| SELECT | WHERE |
|---|---|
| COUNT | is_root |
is_root is an alias for trace.parent_id does not exist, and when used with a WHERE clause, both filter to show only root spans.
If trace.parent_id does not exist is used, Honeycomb automatically updates it to is_root in Query Builder.
Performance Metrics
What are the Slowest Traces in the Application
This query identifies the slowest trace in your application, in terms of duration (duration_ms), and provides information about the specific events and spans that make up that trace.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| MAX(duration_ms) | is_root | name |
- to find potential performance issues
- understand the root cause of slow response times
What is the P90 Duration of Database Calls
This query calculates theP90 duration of your database calls, which is the duration at which 90% of your database calls complete.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| P90(duration_ms) | db.statement exists | db.statement |
- understand the performance of your database
- identify potential issues or bottlenecks
What are the Total Bytes Sent on Requests
This query calculates the total bytes sent on requests, which is the total amount of data that is transmitted by your service in response to requests.HEATMAP creates a histogram data visualization.
Use BubbleUp to further investigate and compare values.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| SUM(http.request_content_length) HEATMAP(http.request_content_length) | http.request_content_length exists | name |
- understand the performance and efficiency of your service
- analyze your data further with BubbleUp
How Much Time Database Calls Take in a Trace
This query provides insight into the performance of your application. If a significant amount of time is spent on database calls, opportunity to improve speed and performance in the way that your application interacts with the database.| SELECT | WHERE | GROUP BY |
|---|---|---|
| SUM(duration_ms) | db.statement exists trace.trace_id = abc123 | db.statement |
What is the Rate at Which the Average Duration for the Service Increases or Decreases
This query calculates the rate of change in the average amount of time that it takes for your service to complete a request.| SELECT | WHERE | GROUP BY |
|---|---|---|
| RATE_AVG(duration_ms) | is_root | name |
- understand the performance of your service
- identify potential trends or patterns in the duration of your service’s requests
Identify Errors Based on API Entry Point
This query uses relational fields to identify errors in the system filtered by top-level API entry point.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | error exists AND meta.annotation_type = span_event AND root.api_entry_point = <api-entry-point> | name |
Find Properties in a Child Span and Group By the Root Span
This query uses relational fields to identify the P95 of database SELECT * statements and find the http.route from which they are being called.| SELECT | WHERE | GROUP BY |
|---|---|---|
| P95(duration_ms) | db.statement starts with select * | root.http.route |
Identify Slow Database Calls for a Service
This query uses relational fields to identify slow database calls for a service.| SELECT | WHERE |
|---|---|
| AVG(duration_ms) | duration_ms > 10000 AND root.service = /cart/checkout |
Error Analysis
Which Exception Happens the Most in the Service
This query provides insight into potential problems in your system. For example, when running a web service that processes online transactions and the most frequent exception being thrown is “TimeoutException”, then your service may have issues connecting to the database to retrieve information about the transactions.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | exception.message exists | exception.message |
Which Spans Contain Which Exceptions
Some OpenTelemetry instrumentation or SDKs will record exception messages on Span Events rather than on the Span itself. This query identifies which Span Events events contain exceptions, and then groups by the name of the Span that corresponds to that Span Event, the name of the service, and then the name of the exception.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | parent_name exists exception.message exists | parent_name service.name exception.message |
Show Only Errored Traces and Their Latency
This query helps to investigate the reasons for errors and latencies. For example, if you know that a errored trace called “XYZ” experiences high latencies, you can take further steps like debugging the code for that trace, adding more resources to handle the workload, or implementing error handling and retry mechanisms.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT HEATMAP(duration_ms) | error = true is_root | name |
Which Tenants Experience Errors on Certain Endpoints
This query identifies which tenants, or specific groups of users or data within a software system, experience endpoint errors. With this knowledge, you can provide specialized support and troubleshooting to help resolve the issue. For example, when running a storage service with multiple tenants, you notice that a particular tenant experiences a high number of errors when accessing their data on your endpoint. This may indicate that a problem exists with their configuration or usage of your service.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | error = true is_root app.tenant exists | app.tenant name |
Identify Errors Based on API Entry Point
This query uses relational fields to identify errors in the system filtered by top-level API entry point.| SELECT | WHERE |
|---|---|
| COUNT | error exists AND root.api_entry_point = api-entry-point |
Identify User Log in Error
This query uses relational fields to identify a user who reported that they cannot log in. Theuser_id is only on the root span, but the error in the auth service exists on a child span within the trace.
| SELECT | WHERE |
|---|---|
| COUNT | root.user_id = <user-id> AND error exists AND service.name = LoginService |
Get a Count of Root Spans with a Child Span that Contains an Error
This query uses relational fields to identify root spans that have a child span that has an error.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | is_root AND any.error exists | name |
Show How Requests to a Service Behave When Another Service in the Trace Experiences a Specific Error
This query uses relational fields to show how requests to a service behave when another service in the trace experiences a specific error.| SELECT | WHERE |
|---|---|
| COUNT | service.name = <service> AND any.service.name = <another-service> AND any.error = <error-name> |
Identify Errors from Database Spans for Traces that Reach a Certain Duration
This query uses relational fields to return the error from database spans for traces longer than a specified duration.| SELECT | WHERE |
|---|---|
| COUNT | root.duration_ms > <duration> AND name = db.statement AND error exists |
Find Spans that Contain Timeout Errors Where Immediate Parent Span Initiated a Database Call
This query uses relational fields to identify spans that contain a timeout error and their immediate parent span initiated a database call.| SELECT | WHERE |
|---|---|
| COUNT | error_code = TimeoutError AND parent.operation = DatabaseCall |
Instrumentation Gap Detection
To detect where an instrumentation gap exists.Identify Missing Field That Should Exist
This query uses relational fields to identify a field that should exist, but currently does not or has no value.| SELECT | WHERE |
|---|---|
| COUNT | <field> does-not-exist AND parent.<field> does-not-exist |
- figure out why a specific field (or set of fields) is missing on several spans.
User Behavior
What is the Number of Requests per Time Period
This query calculates the total number of requests received by your application, and provides a breakdown of those requests by different dimensions, such as the type of request, the endpoint, or the tenant.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | is_root | name |
- monitor the overall traffic and usage of your application
What is the Number of Concurrent Calls for a Specific Span
This query calculates the number of concurrent calls for the individual, specific span namedmy-span, which is the number of calls to that span that are executing simultaneously.
| SELECT | WHERE |
|---|---|
| CONCURRENCY | name = my-span |
- understand the workload and performance of your span
- identify potential issues or bottlenecks that may be affecting the concurrency of your calls
What is the Number of Distinct Users in the Application
This query displays the level of demand on your system and the resources it uses. If the number of distinct users increases over time, it may indicate that your system is approaching its capacity and that action may be needed to improve its performance.| SELECT | WHERE |
|---|---|
| COUNT_DISTINCT(app.user_id) | app.user_id exists |
Endpoint Usage
What is the Load Across Each Server
This query calculates the number of executed, independent traces on each server.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | is_root | host.name |
- understand the workload and performance of your servers
- identify potential issues or bottlenecks
Which Tenant hits Each Endpoint the Most and Their Highest Experienced Latency
The query identifies tenants, or specific groups of users or data within a software system, the endpoints they use, and their maximum experienced latency. For example, if you know that a certain tenant hits endpoint/api/users the most and experiences high latencies on that endpoint, you can investigate the reasons for those latencies and take steps to improve the performance of that endpoint for that tenant.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT MAX(duration_ms) | is_root app.tenant exists | app.tenant name |
High CPU Usage with User Login Request
This query uses relational fields to identify spans with high CPU usage where the root span has an operation of “UserLoginRequest”.| SELECT | WHERE |
|---|---|
| AVG(cpu_usage) | root.operation = UserLoginRequest AND cpu_usage > 0.8 |
Advanced Visualization
Distribution Density of Status Codes Over Time
This query presents the frequency and density of different status codes that are returned by your application during a specified time period.HEATMAP creates a histogram data visualization.
Use BubbleUp to further investigate and compare values.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| HEATMAP(http.status_code) | http.status_code exists | name |
- understand the performance and behavior of your application
- analyze your data further with BubbleUp
Relational Fields
These queries use relational fields for advanced filtering. To learn about more best practices, visit Best Practices for Querying using Relational Fields.root prefix
Identify Errors Based on API Entry Point
This query uses relational fields to identify errors in the system filtered by top-level API entry point.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | error exists AND meta.annotation_type = span_event AND root.api_entry_point = <api-entry-point> | name |
Identify Errors from Database Spans for Traces that Reach a Certain Duration
This query uses relational fields to return the error from database spans for traces longer than a specified duration.| SELECT | WHERE |
|---|---|
| COUNT | root.duration_ms > <duration> AND name = db.statement AND error exists |
Find Properties in a Child Span and Group By the Root Span
This query uses relational fields to identify the P95 of database SELECT * statements and find the http.route from which they are being called.| SELECT | WHERE | GROUP BY |
|---|---|---|
| P95(duration_ms) | db.statement starts with select * | root.http.route |
Identify Slow Database Calls for a Service
This query uses relational fields to identify slow database calls for a service.| SELECT | WHERE |
|---|---|
| AVG(duration_ms) | duration_ms > 10000 AND root.service = /cart/checkout |
Identify Errors Based on API Entry Point
This query uses relational fields to identify errors in the system filtered by top-level API entry point.| SELECT | WHERE |
|---|---|
| COUNT | error exists AND root.api_entry_point = api-entry-point |
Identify User Log in Error
This query uses relational fields to identify a user who reported that they cannot log in. Theuser_id is only on the root span, but the error in the auth service exists on a child span within the trace.
| SELECT | WHERE |
|---|---|
| COUNT | root.user_id = <user-id> AND error exists AND service.name = LoginService |
Identify Errors from Database Spans for Traces that Reach a Certain Duration
This query uses relational fields to return the error from database spans for traces longer than a specified duration.| SELECT | WHERE |
|---|---|
| COUNT | root.duration_ms > <duration> AND name = db.statement AND error exists |
High CPU Usage with User Login Request
This query uses relational fields to identify spans with high CPU usage where the root span has an operation of “UserLoginRequest”.| SELECT | WHERE |
|---|---|
| AVG(cpu_usage) | root.operation = UserLoginRequest AND cpu_usage > 0.8 |
parent prefix
Identify Missing Field That Should Exist
This query uses relational fields to identify a field that should exist, but currently does not or has no value.| SELECT | WHERE |
|---|---|
| COUNT | <field> does-not-exist AND parent.<field> does-not-exist |
- figure out why a specific field (or set of fields) is missing on several spans.
Find Spans that Contain Timeout Errors Where Immediate Parent Span Initiated a Database Call
This query uses relational fields to identify spans that contain a timeout error and their immediate parent span initiated a database call.| SELECT | WHERE |
|---|---|
| COUNT | error_code = TimeoutError AND parent.operation = DatabaseCall |
child prefix
Select P99 Duration for a Span Only When it has Called into Another Specific Span
You have a span namedspanA that sometimes has a child named spanB.
You want to select the P99 of spanA when it has called into spanB.
| SELECT | WHERE | GROUP BY | ORDER BY | LIMIT |
|---|---|---|---|---|
| P99(duration_ms) | name = spanA child.name exists | None | P99(duration_ms) desc | 100 |
Select Sampling of Children for a Given Span
You have a span namedspanA that may call a number of different operations.
You want to see a list/sampling of the functions into which it calls directly.
| SELECT | WHERE | GROUP BY | ORDER BY | LIMIT |
|---|---|---|---|---|
| COUNT | name = spanA | child.name | COUNT desc | 100 |
none prefix
Find All Traces with a Missing Root Span
You want to find all traces that are missing a root span. This can help you identify where your instrumentation may need some troubleshooting.| SELECT | WHERE | GROUP BY | ORDER BY | LIMIT |
|---|---|---|---|---|
| COUNT | none.trace.parent_id does-not-exist | trace.trace_id | COUNT desc | 1000 |
Find Instances of a Process Where No Retry Succeeded
You have a process namedProcessA that may retry multiple times if it fails, before giving up (and for example, executing a rollback).
This means that you might have a few instances of the same pattern repeat in your trace.
If the process succeeds, one of them will also contain a success span named Commit.
You want to find instances where none of the retries succeeded, and examine the root spans that kicked off those instances to explore patterns that might emerge.
| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | name = ProcessA none.name = Commit | trace.trace_id root.name |
anyX prefix
Show How Requests to a Service Behave When Another Service in the Trace Experiences a Specific Error
This query uses relational fields to show how requests to a service behave when another service in the trace experiences a specific error.| SELECT | WHERE |
|---|---|
| COUNT | service.name = <service> AND any.service.name = <another-service> AND any.error = <error-name> |
Get a Count of Root Spans with a Child Span that Contains an Error
This query uses relational fields to identify root spans that have a child span that has an error.| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | is_root AND any.error exists | name |
Find Traces that Contain a Specific 3-way Combination
You do not know your exact trace structure, but you have a hypothesis that a certain function call (functionA) and a certain database transaction (db.rollback) might both be related to some 500 errors that you are seeing.
You want to find traces where:
status_codeis returned as500functionAis called at some point in the tracedb.rollbackis called at some point in the trace
| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | any.response.status_code = 500 any2.name = functionA any3.name = db.rollback | trace.trace_id |
Identify the location of a data/field within a trace
You do not know your exact trace structure, but you want to see customer team names and their device platform alongside some data you are looking at from a service namedServiceA.
Team name and device platform are not available on the spans you are currently targeting, but they may be available on other spans (team.name is available on one specific span, platform is available on another specific span).
| SELECT | WHERE | GROUP BY |
|---|---|---|
| COUNT | service.name = ServiceA any.team.name exists any2.platform exists | any.team.name any2.platform |