To provide additional context and insight into what is happening in your data, you may want to enhance your traces by adding metadata using span annotations.
Adding span annotations makes it easier to understand what’s happening inside a distributed system and lets you troubleshoot performance issues or failures more efficiently.
Annotating Spans
A span annotation is a type of event used within tracing data to add granular observability without creating additional spans.
You can use span annotations to:
Attach additional context to a span at a particular moment in time
Establish relationships between spans that are part of different trace hierarchies
In Honeycomb, span annotations appear as tabbed entries in a selected span’s tracing sidebar (rather than in the trace’s waterfall graph).
Attach Context at a Particular Moment in Time
Span annotations offer a lightweight way to provide more detailed insights into specific moments within a span, which can help you troubleshoot, analyze performance, and improve your overall understanding of complex systems.
A span event is a type of span annotation that lets you attach additional context or metadata to a span at a particular moment in time.
More specifically, you can use span events to record data that correlates with logs, metrics, and other error monitoring tools.
Uses
You may want to use span event annotations to:
Capture significant moments or key milestones.
For example, record the start and end of a sub-operation (such as a database query or external API call), business logic transitions (such as customer authentication or payment verification), or state changes in an application (such as cache hit or miss).
Improve debugging and troubleshooting.
For example, when an error occurs, log the event with detailed information about the failure, such as exception messages, stack traces, or other diagnostic data.
Add more granular observability without creating extra spans.
For example, when processing a request, instead of creating a new span for every minor step, such as reading headers or validating a token, use span events to mark these steps.
Perform time-based analysis.
For example, measure latency between events, the duration of intermediate steps in a span, or the sequence of execution.
Enhance context and logging.
For example, add HTTP status codes at the time of a response or add user IDs relevant to an action.
Correlate with external systems or actions.
For example, record external service responses, notifications received, or security events.
Scenario: Record errors produced inside a loop
Scenario:
You have a span that represents a specific operation in your service.
That operation has a loop, inside of which a non-fatal error could occur.
You want to record any errors that are produced inside of the loop.
Solution:
Attach each error event as a span event annotation on the span that represents the operation.
Doing so will let you capture all of the errors.
Conversely, if you write the error to an error field on the span that represents the operation, then you will overwrite any errors from previous loop iterations already recorded in the field.
Adding Span Events
Important
General rules:
Span events are timestamped, structured events without a duration (aka logs).
Associate span events with a timestamp (just as you do with other events)
Attach span events only to spans that are not regular trace spans.
Span events are not counted towards the total span count for the trace to which they belong.
To add a span event annotation, create an event that includes the following key/value pairs:
Honeycomb Field Name
Description
meta.annotation_type
Type of annotation. To identify an annotation as a span event, set to "span_event".
trace.parent_id
ID of the span to which you want to attach this span event.
trace.trace_id
ID of the trace to which you want this span event to belong.
name
Name of the span event.
service.name (optional)
Name of the service in which the span event occurred.
Establish Relationships Between Spans in Different Trace Hierarchies
Although the general tracing data model, which focuses on the parent-child relationship between spans, can adequately describe most spans, in special cases, you may need to describe a less direct, causal relationship between spans.
A span link is a type of span annotation that lets you establish relationships between spans that are part of different trace hierarchies.
Uses
You may want to use span link annotations to:
Handle asynchronous or parallel operations in distributed systems.
For example, in a service that initiates multiple background tasks (each of which may create its own trace), use span links to associate the traces created by background tasks back to the initiating span.
Correlate related traces across complex systems, in which traces are split across different services or organizations.
For example, when integrating with external APIs or microservices, links spans from different traces to connect requests and responses that exist in separate distributed tracing systems.
Track the relationship between fan-out and fan-in phases.
For example, in a data processing pipeline where different stages run in parallel, use span links to show how the final aggregation step is related to earlier individual processing spans.
Track the flow of work in event-driven architectures where spans may not have direct parent-child relationships.
For example, in a message queue system, where one service produces a message and another service consumes it later, use span links to relate the decoupled producer and consumer spans to give context about the entire message lifecycle.
Accurately model the relationships in complex workflows with multiple initiators.
For example, in a system where multiple services can trigger a batch processing job and each service has its own trace, use span links to link the batch processing job’s span to all of the initiating spans to show how multiple inputs led to the same job.
Merge traces from different requests that logically contribute to the same end result.
For example, if two separate user actions trigger the same downstream process, use span links to link those separate user request traces to the same result span.
Track and correlate tracing and retry/fallback logic.
For example, if an HTTP request fails and the system retries, use span links to connect the retry spans to the original span, so you get full visibility into the retry logic.
Adding Span Links
Important
General rules:
Span links are optional.
Span links can point to spans within the same trace or spans in different traces.
Each span link points to one other span.
Each span can have one or many span links attached to it.
To add a span link annotation, create an event that includes the following key/value pairs:
Honeycomb Field Name
Description
meta.annotation_type
Type of annotation. To identify an annotation as a Span Link, set to "link".
trace.parent_id
ID of the span from which you wish to link. The link will be attached to this span.
trace.trace_id
ID of the trace from which you wish to link. The link will belong to this trace.
