We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Sending trace data to Honeycomb

Your events must contain tracing metadata for Honeycomb to reconstruct and visualize traces for you. There are several ways to generate trace data and send it to Honeycomb:

You can combine multiple types of tracing instrumentation as long as your trace and span IDs are unique. The following sections describe how to use these methods.

Honeycomb Beelines  🔗

Beelines are an easy way to get basic tracing for a single service. Beelines automatically track trace and span relationships using the following fields:

Field Description
name The name of the function or method where the span was created
service_name The name of the Beeline-instrumented service
duration_ms How much time the span took, in milliseconds
trace.span_id The unique ID for each span
trace.trace_id The ID of the trace this span belongs to
trace.parent_id The ID of this span’s parent span, the call location the current span was called from

For more information on installing and configuring Beelines, see Honeycomb Beelines.

OpenTelemetry (beta)  🔗

OpenTelemetry is a vendor-agnostic effort to combine distributed tracing, metrics and logging into a single set of system components and language-specific libraries.

Honeycomb OpenTelemetry Go Exporter  🔗

The Honeycomb OpenTelemetry Go Exporter allows you to gather traces with OpenTelemetry and export them to Honeycomb using Go.

To create the exporter:

  exporter, _ := honeycomb.NewExporter(
      honeycomb.Config{
        APIKey:  <YOUR-API-KEY>,
      },
      honeycomb.TargetingDataset(<YOUR-DATASET>),
      honeycomb.WithServiceName("example-server"),
      honeycomb.WithDebugEnabled()
    ) // optional to output diagnostic logs to STDOUT

  defer exporter.Close()
  sdktrace.NewProvider(sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
      sdktrace.WithSyncer(exporter))

To see an example of the Honeycomb OpenTelemetry Go Exporter in action, try out the OpenTelemetry-Exporter-Go Example App.

OpenTelemetry Collector  🔗

The OpenTelemetry Collector provides a way to receive, process and export tracing data. It can receive spans in a number of open-source formats and export them to one or more destinations.

Honeycomb OpenTelemetry Collector Exporter  🔗

The Honeycomb OpenTelemetry Collector Exporter allows you to export spans received by the collector to Honeycomb.

To configure the exporter in your collector configuration:

exporters:
  honeycomb:
    api_key: "my-api-key"
    dataset: "my-dataset"
    api_url: "https://api.testhost.io"
    sample_rate: 25
    debug: true

OpenCensus  🔗

OpenCensus is a vendor-agnostic single distribution of libraries to provide observability for your systems.

Honeycomb OpenCensus Go Exporter  🔗

The Honeycomb OpenCensus Go Exporter allows you to gather traces and export them to Honeycomb using Go.

To create the exporter:

package main

import (
    honeycomb "github.com/honeycombio/opencensus-exporter/honeycomb"
    "go.opencensus.io/trace"
)

func main() {
    exporter := honeycomb.NewExporter("YOUR-HONEYCOMB-WRITE-KEY", "YOUR-DATASET-NAME")
    defer exporter.Close()

    trace.RegisterExporter(exporter)
}

To see an example of the Honeycomb OpenCensus Go Exporter in action, try out the OpenCensus-Exporter Example App.

Honeycomb OpenCensus Python Exporter  🔗

Python users can use the community-created Honeycomb OpenCensus Python Exporter, by @codeboten.

Honeycomb OpenCensus Elixir Exporter  🔗

Elixir users can use the community-created Honeycomb OpenCensus Elixir Exporter, by @garthk.

OpenTracing  🔗

NOTE: The honeycomb-opentracing-proxy is now deprecated as OpenTracing has been superseded by OpenTelemetry. Please see OpenTelemetry Collector for an alternative.

OpenTracing is a vendor-neutral standard for distributed tracing data. Honeycomb uses the same data format as Zipkin, a common Open Source tracing system.

honeycomb-opentracing-proxy is a lightweight proxy that runs in your infrastructure. This replacement Zipkin collector translates your Zipkin tracing data and sends it to Honeycomb. The proxy can also forward data to your existing collector in addition to Honeycomb.

To see an example of the honeycomb-opentracing-proxy in action, try out the Kubernetes-Envoy-Tracing Example App.

Installing the proxy  🔗

Install honeycomb-opentracing-proxy on a host that can receive your Zipkin tracing data on port 9411. There are two ways to install:

Use a Docker image  🔗

Build from source  🔗

Note: To use honeycomb-opentracing-proxy with Kubernetes, see the README and sample manifest in the honeycomb-opentracing-proxy Github repository. The README contains more detail about configuring the proxy for Kubernetes.

Sending data to the proxy  🔗

Configure your OpenTracing-instrumented applications to use a Zipkin library and send their spans to honeycomb-opentracing-proxy. honeycomb-opentracing-proxy formats the incoming trace data as Honeycomb events and sends them to the Honeycomb API. To also send data to an existing Zipkin collector, use the --downstream option described below.

Configuring the proxy  🔗

To configure honeycomb-opentracing-proxy, specify your desired options on the command line. You must include -d and -k to specify the destination dataset and your Honeycomb API Key. Other flags are optional. The table below describes the required and optional flags for honeycomb-opentracing-proxy:

Option Description Required?
-d dataset The dataset to send traces to Required
-k $YOUR_API_KEY Your Honeycomb API Key Required
--samplerate N Sample by sending 1 of N traces to Honeycomb Optional
--drop_field fieldname Omit the specified field (to remove sensitive data) Optional
--downstream URL:port Forward spans to another Zipkin collector Optional
--debug Write spans to stdout Optional

Note: honeycomb-opentracing-proxy samples by traces rather than spans. For example, -samplerate 10 sends one of every 10 complete traces, including however many spans that selected trace contains. Sampled traces are still complete traces.

A note on Jaeger  🔗

At the time of writing, the Honeycomb OpenTracing Proxy only supports the Zipkin v1 (Thrift) format. In the Data format and transport for reporting spans to Jaeger backend section of the Jaeger docs, you will notice that this is supported for sending from Golang. Based on the Jaeger Java client docs, this approach seems to be supported in Java as well.

To send Zipkin Thrift over HTTP from Jaeger in Golang, create a new Reporter in your Jaeger intialization. Error handling is skipped in this example for brevity.

import (
	"github.com/uber/jaeger-client-go/config"
	"github.com/uber/jaeger-client-go/transport/zipkin"
	jaeger "github.com/uber/jaeger-client-go"
)

cfg, _ := config.FromEnv()

transport, _ := zipkin.NewHTTPTransport(
	"http://honeycomb-opentracing-proxy:9411/api/v1/spans",
	zipkin.HTTPLogger(jaeger.StdLogger),
)

tracer, _, _ := cfg.NewTracer(
	// ... other options ...
	config.Reporter(jaeger.NewRemoteReporter(transport)),
)

Sending data to the Honeycomb OpenTracing Proxy with jaeger.thrift over HTTP or UDP is not yet supported. If you’d like to see support, please notify us via Intercom in the lower right hand corner of the screen.

A note on Istio  🔗

Tracing data emitted by the service mesh Istio can be forwarded to Honeycomb using the OpenTracing Proxy and the proper configuration for Istio.

Istio has configuration options that can be set using the Kubernetes package manager Helm, which generates the final Kubernetes resource definitions used to deploy or update the service mesh. Some of these options are relevant to tracing.

Using Istio with Honeycomb is a three step process:

  1. Add your Honeycomb API key as a Kubernetes secret. e.g.:

    kubectl create secret generic -n default honeycomb-writekey --from-literal=key=$YOUR_WRITE_KEY
    
  2. Install the Honeycomb OpenTracing proxy as a Deployment with corresponding Service in your Kubernetes cluster. e.g.:

    kubectl apply -f https://raw.githubusercontent.com/honeycombio/honeycomb-opentracing-proxy/master/kubernetes/example-manifest.yaml
    
  3. Configure Istio to send the tracing data it generates to the proxy. The Honeycomb proxy “speaks” the Zipkin protocol, so we configure Istio to point at the proxy as if it were the Zipkin collector. For instance, the helm template command accepts Istio configuration options as arguments to the --set flag, so we want to use a command like this:

    helm template --set tracing.enabled=true \
                    --set tracing.provider=zipkin \
                    --set global.tracer.zipkin.address=honeycomb-opentracing-proxy.default.svc.cluster.local:9411 \
                    <usual remaining args>
    

    Once this has been re-generated, kubectl apply the new resulting YAML.

To properly trace all services in your system, you will need to ensure that you are forwarding tracing headers from your apps so that Istio/Envoy can inject the correct tracing information as requests are made and received. See this section on trace context propagation in the Istio docs for details on how to do this.

Manual tracing  🔗

If you have structured logging but aren’t using an OpenTracing-compatible library, add tracing metadata to your existing structured logs. Honeycomb reconstructs your traces from the metadata you provide in your events.

To manually construct tracing metadata, generate unique trace and span IDs and thread them through your applications. Your span, parent span, and trace IDs must accurately reflect the relationships between all the spans that make up a trace. For distributed services, downstream services need trace and span IDs from the services that called them.

Include the following key/value pairs in your log events:

Field Description
name The specific call location (like a function or method name)
trace.span_id A unique ID for each span
trace.parent_id The ID of this span’s parent span, the call location the current span was called from
trace.trace_id The ID of the trace this span belongs to
service_name The name of the service that generated this span
duration_ms How much time the span took, in milliseconds

The trace_id, span_id, and parent_id must come through to Honeycomb as strings. (The strings may be all numeric, but the JSON package should enclose them in quotes.) The root span of a trace is defined by having its parent_id omitted.

Honeycomb also expects all events to contain a timestamp field; if one is not provided, the server will associate the current time of ingest with the given payload. With tracing, this will result in nonsensical waterfall diagrams, with parent spans appearing to start after their child spans have completed.

A root span, the first span in a trace, does not have a parent. As you instrument your code, make sure every span propagates its trace.trace_id and trace.span_id to any child spans it calls, so that the child span can use those values as its trace.trace_id and trace.parent_id. Honeycomb uses these relationships to determine the order spans execute and construct the waterfall diagram.

Send each complete instrumented event after its unit of work finishes. You can do this with honeytail or a Honeycomb SDK. The Honeycomb examples repository on GitHub has instrumentation examples for several languages.

To see an example of manual tracing with Golang in action, try out the Golang-Wiki-Tracing Example App.

To see an example of manual tracing with Ruby in action, try out the Ruby-Wiki-Tracing Example App.

Changing the trace schema  🔗

If you are a team owner and are configuring tracing data sent via Secure Tenancy or need to update the trace schema for a dataset because the data you’re sending has changed field names, you can do this from the Definitions tab for that dataset’s details:

dataset configuration tracing tab

When you’ve updated the field name(s) as desired, click Update. The tracing configuration is updated.

Span Annotations  🔗

Span Annotations are a special kind of event used with tracing data. Span Annotations are visualized differently in Honeycomb, they are not shown in the Waterfall graph. Instead, when you click on a Span that has Annotations, they appear as tabbed entries in the tracing sidebar for that Span.

Honeycomb supports two types of Span Annotations: Span Events and Links.

Span Events  🔗

Span Events are timestamped structured logs (aka events), without a duration. They occur during the course of a Span and can be thought of as annotations on the Span. For example, you might have a Span that represents a specific operation in your service. That operation could have a loop, in which a non-fatal error can occur. If you write the error to an error field on the Span, you’ll overwrite any previous errors (from previous loop iterations) recorded in that field. This is a perfect use case for Span Events, the error events can be attached as Span Event Annotations and you capture all the errors.

For Honeycomb to recognize Span Events, you must tag these events with meta.annotation_type: span_event.

To use Span Events, include the following key/value pairs in your event.

Field Description
meta.annotation_type: "span_event" This combination of key:value is required and allows us to correctly render span events
trace.parent_id The span ID this span event will be attached to
trace.trace_id The ID of the trace this span event belongs to
Timestamp The timestamp at which the span event occurred
name The name of the span event
service_name The name of the service in which the span event occurred (optional)

The field used to tell Honeycomb which field represent Span Annotations can be configured in your dataset’s settings page on the Definitions tab.

A span may be linked to zero or more other spans or traces that are causally related. They can point to another span within the same trace or a span in a different trace. The tracing data model focuses on the parent-child relationship between spans, and most spans can be adequately described with just a span id, a parent span id, and a trace ID. However, in some special cases, it may be useful to describe a less direct causal relationship between spans. Links are optional, but can be useful for expressing a causal relationship to one or more spans or traces elsewhere in your dataset. Links can be used to represent batched operations where a span was initiated by multiple initiating spans, each representing a single incoming item being processed in the batch.

For Honeycomb to recognize Link Annotations, you must tag these events with meta.annotation_type: link.

To use links, include the following key/value pairs in your link event.

Field Description
trace.link.span_id The span ID you wish to link to
trace.link.trace_id The trace ID you wish to link to
meta.annotation_type: "link" This combination of key:value is required and allows us to correctly render links
trace.parent_id The span ID this link will be attached to
trace.trace_id The ID of the trace this link belongs to

You can tell Honeycomb which fields you would like used for these in you dataset’s settings on the Definitions tab.

Next steps  🔗

Now that you are generating tracing data, view your traces and spans in Honeycomb. See Exploring trace data to learn more about how to query your tracing data.