We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Sending trace data to Honeycomb

Your events must contain tracing metadata for Honeycomb to reconstruct and visualize traces for you. There are several ways to generate trace data and send it to Honeycomb:

You can combine multiple types of tracing instrumentation as long as your trace and span IDs are unique. The following sections describe how to use these methods.

OpenTracing

OpenTracing is a vendor-neutral standard for distributed tracing data. Honeycomb uses the same data format as Zipkin, a common Open Source tracing system that works with OpenTracing.

honeycomb-opentracing-proxy is a lightweight proxy that runs in your infrastructure. This replacement Zipkin collector translates your Zipkin-compatible tracing data and sends it to Honeycomb. The proxy can also forward data to your existing collector in addition to Honeycomb.

To see an example of the honeycomb-opentracing-proxy in action, try out the Kubernetes-Envoy-Tracing Example App.

Installing the proxy

Install honeycomb-opentracing-proxy on a host that can receive your Zipkin-compatible tracing data on port 9411. There are two ways to install:

Use a Docker image

Build from source

You are currently logged in to the team, so we have populated the write key here to the first write key for that team.

Note: To use honeycomb-opentracing-proxy with Kubernetes, see the README and sample manifest in the honeycomb-opentracing-proxy Github repository. The README contains more detail about configuring the proxy for Kubernetes.

Sending data to the proxy

Configure your OpenTracing-instrumented applications to use a Zipkin library and send their spans to honeycomb-opentracing-proxy. honeycomb-opentracing-proxy formats the incoming trace data as Honeycomb events and sends them to the Honeycomb API. To also send data to an existing Zipkin collector, use the --downstream option described below.

Configuring the proxy

To configure honeycomb-opentracing-proxy, specify your desired options on the command line. You must include -d and -k to specify the destination dataset and your Honeycomb API Key. Other flags are optional. The table below describes the required and optional flags for honeycomb-opentracing-proxy:

Option Description Required?
-d dataset The dataset to send traces to Required
-k $YOUR_API_KEY Your Honeycomb API Key Required
--samplerate N Sample by sending 1 of N traces to Honeycomb Optional
--drop_field fieldname Omit the specified field (to remove sensitive data) Optional
--downstream URL:port Forward spans to another Zipkin collector Optional
--debug Write spans to stdout Optional

Note: honeycomb-opentracing-proxy samples by traces rather than spans. For example, -samplerate 10 sends one of every 10 complete traces, including however many spans that selected trace contains. Sampled traces are still complete traces.

Honeycomb Beelines

Beelines are an easy way to get basic tracing for a single service. Beelines automatically track trace and span relationships using the following fields:

Field Description
name The name of the function or method where the span was created
service_name The name of the Beeline-instrumented service
duration_ms How much time the span took, in milliseconds
trace.span_id The unique ID for each span
trace.trace_id The ID of the trace this span belongs to
trace.parent_id The ID of this span’s parent span, the call location the current span was called from

Note: A Honeycomb Beeline generates tracing data for the service where it is installed. To trace spans across multiple services, instrument your code for tracing using another method. You can still use Beelines even if you use other instrumentation for distributed tracing.

For more information on installing and configuring Beelines, see Honeycomb Beelines.

Honeycomb OpenCensus Go Exporter

OpenCensus is a vendor-agnostic single distribution of libraries to provide observability for your systems. The Honeycomb OpenCensus Go Exporter allows you to gather traces and export them to Honeycomb.

Creating the Exporter

To create the exporter:

package main

import (
    honeycomb "github.com/honeycombio/opencensus-exporter/honeycomb"
    "go.opencensus.io/trace"
)

func main() {
    exporter := honeycomb.NewExporter("YOUR-HONEYCOMB-WRITE-KEY", "YOUR-DATASET-NAME")
    defer exporter.Close()

    trace.RegisterExporter(exporter)
}

To see an example of the Honeycomb OpenCensus Go Exporter in action, try out the OpenCensus-Exporter Example App.

Manual tracing

If you have structured logging but aren’t using an OpenTracing-compatible library, add tracing metadata to your existing structured logs. Honeycomb reconstructs your traces from the metadata you provide in your events.

To manually construct tracing metadata, generate unique trace and span IDs and thread them through your applications. Your span, parent span, and trace IDs must accurately reflect the relationships between all the spans that make up a trace. For distributed services, downstream services need trace and span IDs from the services that called them.

Include the following key/value pairs in your log events:

Field Description
name The specific call location (like a function or method name)
id A unique ID for each span
parentId The ID of this span’s parent span, the call location the current span was called from
traceId The ID of the trace this span belongs to
serviceName The name of the service that generated this span
durationMs How much time the span took, in milliseconds

Make sure the timestamp for the event corresponds to the start of the span. If left off, the API server will assign the timestamp when it receives the event. This will result in nonsensical waterfall diagrams, with parent spans appearing to start after their child spans have completed.

A root span, the first span in a trace, does not have a parent. As you instrument your code, make sure every span propagates its traceId and id to any child spans it calls. Honeycomb uses these values to determine the order spans execute and construct the waterfall diagram.

Send each complete instrumented event after its unit of work finishes. You can do this with honeytail or a Honeycomb SDK. The Honeycomb examples repository on GitHub has instrumentation examples for several languages.

To see an example of manual tracing with Golang in action, try out the Golang-Wiki-Tracing Example App.

To see an example of manual tracing with Ruby in action, try out the Ruby-Wiki-Tracing Example App.

Next steps

Now that you are generating tracing data, view your traces and spans in Honeycomb. See Exploring trace data to learn more about how to query your tracing data.