> ## Documentation Index > Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt > Use this file to discover all available pages before exploring further. # Examples: Query for Traces > Copy and adapt example trace queries to analyze the performance and behavior of your application using Honeycomb's Query Builder. Use the query examples below to explore the performance and behavior of your application. The specific attributes below may or may not exist in your data and environment. Enter each example query using the [Query Builder](/investigate/query/build/). These example queries use two to three of the `SELECT`, `WHERE`, and `GROUP BY` clauses, located at the top of the Query Builder. * `SELECT` - Performs a calculation and displays a corresponding graph over time. Most `SELECT` queries return a line graph while the `HEATMAP` visualization shows the distribution of data over time * `WHERE` - Filters based on attribute parameter(s) * `GROUP BY` - Groups fields by attribute parameter(s) Screenshot of Select, Where, and Group by clauses in Query Builder ## Number of Total Root Spans This query calculates the total number of root spans received by your application by looking for all spans without a parent span ID. | SELECT | WHERE | | ------ | -------- | | COUNT | is\_root | `is_root` is an alias for `trace.parent_id does not exist`, and when used with a **WHERE** clause, both filter to show only root spans. If `trace.parent_id does not exist` is used, Honeycomb automatically updates it to `is_root` in Query Builder. ## Performance Metrics ### What are the Slowest Traces in the Application This query identifies the slowest trace in your application, in terms of duration (`duration_ms`), and provides information about the specific events and spans that make up that trace. | SELECT | WHERE | GROUP BY | | ----------------- | -------- | -------- | | MAX(duration\_ms) | is\_root | name | Use to: * to find potential performance issues * understand the root cause of slow response times ### What is the P90 Duration of Database Calls This query calculates the `P90` duration of your database calls, which is the duration at which 90% of your database calls complete. | SELECT | WHERE | GROUP BY | | ----------------- | ------------------- | ------------ | | P90(duration\_ms) | db.statement exists | db.statement | Use to: * understand the performance of your database * identify potential issues or bottlenecks ### What are the Total Bytes Sent on Requests This query calculates the total bytes sent on requests, which is the total amount of data that is transmitted by your service in response to requests. `HEATMAP` creates a histogram data visualization. Use [BubbleUp](/investigate/analyze/identify-outliers/) to further investigate and compare values. | SELECT | WHERE | GROUP BY | | -------------------------------------------------------------------------------- | ------------------------------------ | -------- | | SUM(http.request\_content\_length)
HEATMAP(http.request\_content\_length) | http.request\_content\_length exists | name | Use to: * understand the performance and efficiency of your service * analyze your data further with BubbleUp ### How Much Time Database Calls Take in a Trace This query provides insight into the performance of your application. If a significant amount of time is spent on database calls, opportunity to improve speed and performance in the way that your application interacts with the database. | SELECT | WHERE | GROUP BY | | ----------------- | --------------------------------------------------- | ------------ | | SUM(duration\_ms) | db.statement exists
trace.trace\_id = abc123 | db.statement | ### What is the Rate at Which the Average Duration for the Service Increases or Decreases This query calculates the rate of change in the average amount of time that it takes for your service to complete a request. | SELECT | WHERE | GROUP BY | | ----------------------- | -------- | -------- | | RATE\_AVG(duration\_ms) | is\_root | name | Use to: * understand the performance of your service * identify potential trends or patterns in the duration of your service's requests ### Identify Errors Based on API Entry Point This query uses [relational fields](/investigate/query/build/#relational-fields) to identify errors in the system filtered by top-level API entry point. | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------------------------------------------------------------------------------------- | -------- | | COUNT | error exists AND
meta.annotation\_type = span\_event AND
root.api\_entry\_point = `` | name | ### Find Properties in a Child Span and Group By the Root Span This query uses [relational fields](/investigate/query/build/#relational-fields) to identify the P95 of database SELECT \* statements and find the http.route from which they are being called. | SELECT | WHERE | GROUP BY | | ----------------- | ---------------------------------- | --------------- | | P95(duration\_ms) | db.statement starts with select \* | root.http.route | ### Identify Slow Database Calls for a Service This query uses [relational fields](/investigate/query/build/#relational-fields) to identify slow database calls for a service. | SELECT | WHERE | | ----------------- | ------------------------------------------------------------- | | AVG(duration\_ms) | duration\_ms > 10000 AND
root.service = /cart/checkout | ## Error Analysis ### Which Exception Happens the Most in the Service This query provides insight into potential problems in your system. For example, when running a web service that processes online transactions and the most frequent exception being thrown is "TimeoutException", then your service may have issues connecting to the database to retrieve information about the transactions. | SELECT | WHERE | GROUP BY | | ------ | ------------------------ | ----------------- | | COUNT | exception.message exists | exception.message | ### Which Spans Contain Which Exceptions Some OpenTelemetry instrumentation or SDKs will record exception messages on Span Events rather than on the Span itself. This query identifies which Span Events events contain exceptions, and then groups by the name of the Span that corresponds to that Span Event, the name of the service, and then the name of the exception. | SELECT | WHERE | GROUP BY | | ------ | --------------------------------------------------- | --------------------------------------------------------- | | COUNT | parent\_name exists
exception.message exists | parent\_name
service.name
exception.message | ### Show Only Errored Traces and Their Latency This query helps to investigate the reasons for errors and latencies. For example, if you know that a errored trace called "XYZ" experiences high latencies, you can take further steps like debugging the code for that trace, adding more resources to handle the workload, or implementing error handling and retry mechanisms. | SELECT | WHERE | GROUP BY | | ---------------------------------- | ---------------------------- | -------- | | COUNT
HEATMAP(duration\_ms) | error = true
is\_root | name | ### Which Tenants Experience Errors on Certain Endpoints This query identifies which tenants, or specific groups of users or data within a software system, experience endpoint errors. With this knowledge, you can provide specialized support and troubleshooting to help resolve the issue. For example, when running a storage service with multiple tenants, you notice that a particular tenant experiences a high number of errors when accessing their data on your endpoint. This may indicate that a problem exists with their configuration or usage of your service. | SELECT | WHERE | GROUP BY | | ------ | ----------------------------------------------------- | ---------------------- | | COUNT | error = true
is\_root
app.tenant exists | app.tenant
name | ### Identify Errors Based on API Entry Point This query uses [relational fields](/investigate/query/build/#relational-fields) to identify errors in the system filtered by top-level API entry point. | SELECT | WHERE | | ------ | ---------------------------------------------------------------- | | COUNT | error exists AND
root.api\_entry\_point = api-entry-point | ### Identify User Log in Error This query uses [relational fields](/investigate/query/build/#relational-fields) to identify a user who reported that they cannot log in. The `user_id` is only on the root span, but the error in the auth service exists on a child span within the trace. | SELECT | WHERE | | ------ | ------------------------------------------------------------------------------------------ | | COUNT | root.user\_id = `` AND
error exists AND
service.name = LoginService | ### Get a Count of Root Spans with a Child Span that Contains an Error This query uses [relational fields](/investigate/query/build/#relational-fields) to identify root spans that have a child span that has an error. | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------ | -------- | | COUNT | is\_root AND
any.error exists | name | ### Show How Requests to a Service Behave When Another Service in the Trace Experiences a Specific Error This query uses [relational fields](/investigate/query/build/#relational-fields) to show how requests to a service behave when another service in the trace experiences a specific error. | SELECT | WHERE | | ------ | ------------------------------------------------------------------------------------------------------------------ | | COUNT | service.name = `` AND
any.service.name = `` AND
any.error = `` | ### Identify Errors from Database Spans for Traces that Reach a Certain Duration This query uses [relational fields](/investigate/query/build/#relational-fields) to return the error from database spans for traces longer than a specified duration. | SELECT | WHERE | | ------ | --------------------------------------------------------------------------------------- | | COUNT | root.duration\_ms > `` AND
name = db.statement AND
error exists | ### Find Spans that Contain Timeout Errors Where Immediate Parent Span Initiated a Database Call This query uses [relational fields](/investigate/query/build/#relational-fields) to identify spans that contain a timeout error and their immediate parent span initiated a database call. | SELECT | WHERE | | ------ | --------------------------------------------------------------------- | | COUNT | error\_code = TimeoutError AND
parent.operation = DatabaseCall | ## Instrumentation Gap Detection To detect where an instrumentation gap exists. ### Identify Missing Field That Should Exist This query uses [relational fields](/investigate/query/build/#relational-fields) to identify a field that should exist, but currently does not or has no value. | SELECT | WHERE | | ------ | ------------------------------------------------------------ | | COUNT | `` does-not-exist AND parent.`` does-not-exist | Use to: * figure out why a specific field (or set of fields) is missing on several spans. ## User Behavior ### What is the Number of Requests per Time Period This query calculates the total number of requests received by your application, and provides a breakdown of those requests by different dimensions, such as the type of request, the endpoint, or the tenant. | SELECT | WHERE | GROUP BY | | ------ | -------- | -------- | | COUNT | is\_root | name | Use to: * monitor the overall traffic and usage of your application ### What is the Number of Concurrent Calls for a Specific Span This query calculates the number of concurrent calls for the individual, specific span named `my-span`, which is the number of calls to that span that are executing simultaneously. | SELECT | WHERE | | ----------- | -------------- | | CONCURRENCY | name = my-span | Use to: * understand the workload and performance of your span * identify potential issues or bottlenecks that may be affecting the concurrency of your calls ### What is the Number of Distinct Users in the Application This query displays the level of demand on your system and the resources it uses. If the number of distinct users increases over time, it may indicate that your system is approaching its capacity and that action may be needed to improve its performance. | SELECT | WHERE | | ----------------------------- | ------------------- | | COUNT\_DISTINCT(app.user\_id) | app.user\_id exists | ## Endpoint Usage ### What is the Load Across Each Server This query calculates the number of executed, independent traces on each server. | SELECT | WHERE | GROUP BY | | ------ | -------- | --------- | | COUNT | is\_root | host.name | Use to: * understand the workload and performance of your servers * identify potential issues or bottlenecks ### Which Tenant hits Each Endpoint the Most and Their Highest Experienced Latency The query identifies tenants, or specific groups of users or data within a software system, the endpoints they use, and their maximum experienced latency. For example, if you know that a certain tenant hits endpoint `/api/users` the most and experiences high latencies on that endpoint, you can investigate the reasons for those latencies and take steps to improve the performance of that endpoint for that tenant. | SELECT | WHERE | GROUP BY | | ------------------------------ | --------------------------------- | ---------------------- | | COUNT
MAX(duration\_ms) | is\_root
app.tenant exists | app.tenant
name | ### High CPU Usage with User Login Request This query uses [relational fields](/investigate/query/build/#relational-fields) to identify spans with high CPU usage where the root span has an operation of "UserLoginRequest". | SELECT | WHERE | | --------------- | ------------------------------------------------------------- | | AVG(cpu\_usage) | root.operation = UserLoginRequest AND
cpu\_usage > 0.8 | ## Advanced Visualization ### Distribution Density of Status Codes Over Time This query presents the frequency and density of different status codes that are returned by your application during a specified time period. `HEATMAP` creates a histogram data visualization. Use [BubbleUp](/investigate/analyze/identify-outliers/) to further investigate and compare values. | SELECT | WHERE | GROUP BY | | -------------------------- | ------------------------ | -------- | | HEATMAP(http.status\_code) | http.status\_code exists | name | Use to: * understand the performance and behavior of your application * analyze your data further with BubbleUp ## Relational Fields These queries use [relational fields](/investigate/query/build/#relational-fields) for advanced filtering. To learn about more best practices, visit [Best Practices for Querying using Relational Fields](/get-started/best-practices/relational-fields/). ### `root` prefix #### Identify Errors Based on API Entry Point This query uses [relational fields](/investigate/query/build/#relational-fields) to identify errors in the system filtered by top-level API entry point. | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------------------------------------------------------------------------------------- | -------- | | COUNT | error exists AND
meta.annotation\_type = span\_event AND
root.api\_entry\_point = `` | name | #### Identify Errors from Database Spans for Traces that Reach a Certain Duration This query uses [relational fields](/investigate/query/build/#relational-fields) to return the error from database spans for traces longer than a specified duration. | SELECT | WHERE | | ------ | --------------------------------------------------------------------------------------- | | COUNT | root.duration\_ms > `` AND
name = db.statement AND
error exists | #### Find Properties in a Child Span and Group By the Root Span This query uses [relational fields](/investigate/query/build/#relational-fields) to identify the P95 of database SELECT \* statements and find the http.route from which they are being called. | SELECT | WHERE | GROUP BY | | ----------------- | ---------------------------------- | --------------- | | P95(duration\_ms) | db.statement starts with select \* | root.http.route | #### Identify Slow Database Calls for a Service This query uses [relational fields](/investigate/query/build/#relational-fields) to identify slow database calls for a service. | SELECT | WHERE | | ----------------- | ------------------------------------------------------------- | | AVG(duration\_ms) | duration\_ms > 10000 AND
root.service = /cart/checkout | #### Identify Errors Based on API Entry Point This query uses [relational fields](/investigate/query/build/#relational-fields) to identify errors in the system filtered by top-level API entry point. | SELECT | WHERE | | ------ | ---------------------------------------------------------------- | | COUNT | error exists AND
root.api\_entry\_point = api-entry-point | #### Identify User Log in Error This query uses [relational fields](/investigate/query/build/#relational-fields) to identify a user who reported that they cannot log in. The `user_id` is only on the root span, but the error in the auth service exists on a child span within the trace. | SELECT | WHERE | | ------ | ------------------------------------------------------------------------------------------ | | COUNT | root.user\_id = `` AND
error exists AND
service.name = LoginService | #### Identify Errors from Database Spans for Traces that Reach a Certain Duration This query uses [relational fields](/investigate/query/build/#relational-fields) to return the error from database spans for traces longer than a specified duration. | SELECT | WHERE | | ------ | --------------------------------------------------------------------------------------- | | COUNT | root.duration\_ms > `` AND
name = db.statement AND
error exists | #### High CPU Usage with User Login Request This query uses [relational fields](/investigate/query/build/#relational-fields) to identify spans with high CPU usage where the root span has an operation of "UserLoginRequest". | SELECT | WHERE | | --------------- | ------------------------------------------------------------- | | AVG(cpu\_usage) | root.operation = UserLoginRequest AND
cpu\_usage > 0.8 | ### `parent` prefix #### Identify Missing Field That Should Exist This query uses [relational fields](/investigate/query/build/#relational-fields) to identify a field that should exist, but currently does not or has no value. | SELECT | WHERE | | ------ | ------------------------------------------------------------ | | COUNT | `` does-not-exist AND parent.`` does-not-exist | Use to: * figure out why a specific field (or set of fields) is missing on several spans. #### Find Spans that Contain Timeout Errors Where Immediate Parent Span Initiated a Database Call This query uses [relational fields](/investigate/query/build/#relational-fields) to identify spans that contain a timeout error and their immediate parent span initiated a database call. | SELECT | WHERE | | ------ | --------------------------------------------------------------------- | | COUNT | error\_code = TimeoutError AND
parent.operation = DatabaseCall | ### `child` prefix #### Select P99 Duration for a Span Only When it has Called into Another Specific Span You have a span named `spanA` that sometimes has a child named `spanB`. You want to select the P99 of `spanA` when it has called into `spanB`. | SELECT | WHERE | GROUP BY | ORDER BY | LIMIT | | ----------------- | ------------------------------------- | -------- | ---------------------- | ----- | | P99(duration\_ms) | name = spanA
child.name exists | None | P99(duration\_ms) desc | 100 | #### Select Sampling of Children for a Given Span You have a span named `spanA` that may call a number of different operations. You want to see a list/sampling of the functions into which it calls directly. | SELECT | WHERE | GROUP BY | ORDER BY | LIMIT | | ------ | ------------ | ---------- | ---------- | ----- | | COUNT | name = spanA | child.name | COUNT desc | 100 | ### `none` prefix #### Find All Traces with a Missing Root Span You want to find all traces that are missing a root span. This can help you identify where your instrumentation may need some troubleshooting. | SELECT | WHERE | GROUP BY | ORDER BY | LIMIT | | ------ | ------------------------------------ | --------------- | ---------- | ----- | | COUNT | none.trace.parent\_id does-not-exist | trace.trace\_id | COUNT desc | 1000 | #### Find Instances of a Process Where No Retry Succeeded You have a process named `ProcessA` that may retry multiple times if it fails, before giving up (and for example, executing a rollback). This means that you might have a few instances of the same pattern repeat in your trace. If the process succeeds, one of them will also contain a success span named `Commit`. You want to find instances where none of the retries succeeded, and examine the root spans that kicked off those instances to explore patterns that might emerge. | SELECT | WHERE | GROUP BY | | ------ | ----------------------------------------- | -------------------------------- | | COUNT | name = ProcessA
none.name = Commit | trace.trace\_id
root.name | ### `anyX` prefix #### Show How Requests to a Service Behave When Another Service in the Trace Experiences a Specific Error This query uses [relational fields](/investigate/query/build/#relational-fields) to show how requests to a service behave when another service in the trace experiences a specific error. | SELECT | WHERE | | ------ | ------------------------------------------------------------------------------------------------------------------ | | COUNT | service.name = `` AND
any.service.name = `` AND
any.error = `` | #### Get a Count of Root Spans with a Child Span that Contains an Error This query uses [relational fields](/investigate/query/build/#relational-fields) to identify root spans that have a child span that has an error. | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------ | -------- | | COUNT | is\_root AND
any.error exists | name | #### Find Traces that Contain a Specific 3-way Combination You do not know your exact trace structure, but you have a hypothesis that a certain function call (`functionA`) and a certain database transaction (`db.rollback`) might both be related to some `500` errors that you are seeing. You want to find traces where: * `status_code` is returned as `500` * `functionA` is called at some point in the trace * `db.rollback` is called at some point in the trace | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------------------------------------------------------------- | --------------- | | COUNT | any.response.status\_code = 500
any2.name = functionA
any3.name = db.rollback | trace.trace\_id | #### Identify the location of a data/field within a trace You do not know your exact trace structure, but you want to see customer team names and their device platform alongside some data you are looking at from a service named `ServiceA`. Team name and device platform are not available on the spans you are currently targeting, but they may be available on other spans (`team.name` is available on one specific span, `platform` is available on another specific span). | SELECT | WHERE | GROUP BY | | ------ | ------------------------------------------------------------------------------- | ---------------------------------- | | COUNT | service.name = ServiceA
any.team.name exists
any2.platform exists | any.team.name
any2.platform |