Honeycomb’s data ingest path supports several types of instrumentation, including two approaches specific to instrumenting code: vendor-neutral OpenTelemetry and Honeycomb’s Beelines. Both of these approaches generate the structured event data format that will help you get the most out of Honeycomb.
New to instrumenting code? Want to learn about structured events? Read more about instrumentation.
OpenTelemetry is a cross-industry standard that is supported by many vendors, including Honeycomb. Data generated by OpenTelemetry can be used with Honeycomb and other OpenTelemetry-compatible tools.
The Honeycomb Beelines were written before OpenTelemetry. They use a format and API that can only be used with Honeycomb.
Honeycomb embraces OpenTelemetry as the effective way to instrument applications. For any new observability efforts, we recommend instrumenting with OpenTelemetry.
If you already use Beelines, you can continue to use them. However, we do recommend migrating to OpenTelemetry. You do not need to rewrite all your instrumentation code up front to use OpenTelemetry. If one service is instrumented with OpenTelemetry and another with Beelines, you can still get trace visualizations that include them both.
If you are already using Beelines to instrument some services, but need to instrument a new service, you do not need to continue with Beelines. You can instrument any new services with OpenTelemetry. We call this a Mixed Environment, and you can get trace visualizations from both Beeline and OpenTelemetry instrumentation.
Some things to keep in mind:
Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the W3C Trace Context specification.
To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers.
The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used.
To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services.
Note: Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed.
To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration.
Here is an example of this configuration in the Go Beeline.
To learn more about ongoing development for W3C trace context propagation, visit the W3C Trace Context GitHub repository.
If you instrument each service or process with OpenTelemetry or a Beeline, then your data will be coherent.
The granularity with which you can mix Beelines and OpenTelemetry without additional, more advanced work, is at the process- or service-level. Do not mix them in the same process, as this will result in bad data unless you adopt a wrapper and a strategy for incrementally switching your instrumentation calls to OpenTelemetry.
The names of fields differ on events produced by the Beeline instrumentations and the OpenTelemetry instrumentations.
For example, the Beelines'
service_name field has an equivalent of
service.name in OpenTelemetry events.
To use a single column for this value in queries for the name of a service, define a derived column in the dataset and use the
COALESCE function to determine the name:
COALESCE($service.name, $service_name, "unknown")
IF( EXISTS($service.name), $service.name, EXISTS($service_name), $service_name, "unknown" )
In order to achieve trace-aware sampling across a mixed environment, all instrumented services need to send data to Honeycomb via Refinery.
Do you still have questions about OpenTelemetry and Honeycomb, or have a specific use case for your architecture?
The Honeycomb Pollinators Community Slack is a great place to ask your questions and share your experience. If you misplaced your invite, ask us via chat or email firstname.lastname@example.org and we will send you a new one.