Sending Trace Data to Honeycomb | Honeycomb

Sending Trace Data to Honeycomb

Your events must contain tracing metadata for Honeycomb to reconstruct and visualize traces for you. There are several ways to generate trace data and send it to Honeycomb:

  • Instrument a service using an OpenTelemetry SDK and send it to Honeycomb. This is the preferred way to send trace data to Honeycomb.
  • Send already instrumented open-source tracing metadata to the OpenTelemetry Collector and send it to Honeycomb. This method is best if you already use Zipkin, Jaeger, OpenCensus, OpenTracing, or another compatible distributed tracing system.
  • Instrument a single service using a Honeycomb Beeline. However, because Beelines are proprietary and pre-date OpenTelemetry, we recommend starting with OpenTelemetry if possible, and migrating to OpenTelemetry if you are using a Beeline today.
  • Manually instrument your code to add tracing-specific fields, which the Honeycomb UI can pick up. This method is useful for limited tracing of a few services.

You can combine multiple types of tracing instrumentation as long as your trace and span IDs are unique. The following sections describe how to use these methods.

OpenTelemetry 

OpenTelemetry is a vendor-agnostic effort to combine distributed tracing, metrics, and logging into a single set of system components and language-specific libraries.

Refer to the instructions on using OpenTelemetry to automatically instrument for tracing data.

OpenTelemetry automatically tracks trace and span relationships using the following fields:

Field Description
name The name of the function or method where the span was created
service.name The name of the instrumented service
duration_ms How much time the span took, in milliseconds
trace.span_id The unique ID for each span
trace.trace_id The ID of the trace this span belongs to
trace.parent_id The ID of this span’s parent span, the call location the current span was called from

Honeycomb Beelines 

For more information on installing and configuring Beelines, see Honeycomb Beelines. Note that Beelines create a field named service_name rather than service.name.

Because Beelines are proprietary and pre-date OpenTelemetry, we recommend starting with OpenTelemetry if possible, and migrating to OpenTelemetry if you are using a Beeline today.

Manual Tracing 

If you have structured logging but are not using an OpenTelemetry-compatible library, add tracing metadata to your existing structured logs. Honeycomb reconstructs your traces from the metadata you provide in your events.

To manually construct tracing metadata, generate unique trace and span IDs and thread them through your applications. Your span, parent span, and trace IDs must accurately reflect the relationships between all the spans that make up a trace. For distributed services, downstream services need trace and span IDs from the services that called them.

You should manually include the same key/value pairs in your log events as in the table above. The trace_id, span_id, and parent_id must come through to Honeycomb as strings. (The strings may be all numeric, but the JSON package should enclose them in quotes.) The root span of a trace is defined by having its parent_id omitted.

Honeycomb also expects all events to contain a timestamp field. If a timestamp is not provided, the server will associate the current time of ingest with the given payload. With tracing, this will result in nonsensical waterfall diagrams, with parent spans appearing to start after their child spans have completed.

  • When adding a timestamp, make sure it corresponds to the start of the span.
  • If you are using our Event API, set the timestamp with the X-Honeycomb-Event-Time header. The timestamp itself should be in RFC3339 high precision format (for example, YYYY-MM-DDTHH:MM:SS.mmmZ) or a unix epoch timestamp (seconds since 1970) with second or greater precision (for example, 1452759330927).
  • If you are using Honeytail, you can set the timestamp in a timestamp field. More on that here.

A root span, the first span in a trace, does not have a parent. As you instrument your code, make sure every span propagates its trace.trace_id and trace.span_id to any child spans it calls, so that the child span can use those values as its trace.trace_id and trace.parent_id. Honeycomb uses these relationships to determine the order spans execute and construct the waterfall diagram.

Send each complete instrumented event after its unit of work finishes. You can do this with Honeytail or a Honeycomb SDK. The Honeycomb examples repository on GitHub has instrumentation examples for several languages.

See an example of manual tracing with Golang with the Wiki Tracing Example App using Libhoney Go.

See an example of manual tracing with Ruby with the Wiki Tracing Example App using Libhoney Ruby

Defining Services 

Note: Service scoping is supported only for Honeycomb datasets, not Honeycomb Classic datasets. Learn more about Honeycomb versus Honeycomb Classic datasets.

Service datasets are created from the value of your configured service name. Traces are sent to services datasets where you can query across a single service or across an entire environment. If a service.name is unavailable for your event, it will be sent to an unknown_service dataset where you can troubleshoot your instrumentation.

Changing the Trace Schema 

If you are a team owner and need to update the trace schema for a dataset because the data you are sending has changed field names, you can do this from the Definitions tab for that dataset’s details:

dataset configuration tracing tab

When you have updated the field name(s) as desired, select Update. The tracing configuration is updated.

Span Annotations 

Span Annotations are a special kind of event used with tracing data. Span Annotations are visualized differently in Honeycomb, they are not shown in the Waterfall graph. Instead, when you select a Span that has Annotations, they appear as tabbed entries in the tracing sidebar for that Span.

Honeycomb supports two types of Span Annotations: Span Events and Links.

Span Events 

Span Events are timestamped structured logs (aka events), without a duration. They occur during the course of a Span and can be thought of as annotations on the Span. For example, you might have a Span that represents a specific operation in your service. That operation could have a loop, in which a non-fatal error can occur. If you write the error to an error field on the Span, you will overwrite any previous errors (from previous loop iterations) recorded in that field. This is a perfect use case for Span Events, the error events can be attached as Span Event Annotations and you capture all the errors.

For Honeycomb to recognize Span Events, you must tag these events with meta.annotation_type: span_event. Do not attach this field to regular trace spans. Any span that contains this field will not be counted towards the total span count for the trace.

To use Span Events, include the following key/value pairs in your event.

Field Description
meta.annotation_type Must be "span_event"
trace.parent_id The span ID this span event will be attached to
trace.trace_id The ID of the trace this span event belongs
name The name of the span
service.name The name of the service in which the span event occurred (optional)

As with other events, Span Events should be associated with a timestamp.

The field used to tell Honeycomb which field represent Span Annotations can be configured in your dataset’s settings page on the Definitions tab.

A span may be linked to zero or more other spans or traces that are causally related. They can point to another span within the same trace or a span in a different trace. The tracing data model focuses on the parent-child relationship between spans, and most spans can be adequately described with just a span id, a parent span id, and a trace ID. However, in some special cases, it may be useful to describe a less direct causal relationship between spans. Links are optional, but can be useful for expressing a causal relationship to one or more spans or traces elsewhere in your dataset. Links can be used to represent batched operations where a span was initiated by multiple initiating spans, each representing a single incoming item being processed in the batch.

For Honeycomb to recognize Link Annotations, you must tag these events with meta.annotation_type: link.

To use links, include the following key/value pairs in your link event:

Field Description
meta.annotation_type Must be "link"
trace.parent_id The span ID this link will be attached to
trace.trace_id The ID of the trace this link belongs to
trace.link.span_id The span ID you wish to link to
trace.link.trace_id The trace ID you wish to link to

You can tell Honeycomb which fields you would like used for these in you dataset’s settings on the Definitions tab.

Next Steps 

Now that you are generating tracing data, view your traces and spans in Honeycomb. See Exploring trace data to learn more about how to query your tracing data.

Did you find what you were looking for?