OpenTelemetry for Go | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

OpenTelemetry for Go

This guide will help you add OpenTelemetry to your Go application, show you how to add additional custom context to that instrumentation, and ensure that instrumentation data is being sent to Honeycomb.

Requirements  🔗

These instructions will explain how to set up automatic and manual instrumentation for a service written in Go. In order to follow along, you will need:

  • Go 1.14
  • An application written in Go
  • A Honeycomb API Key. You can find your API key on your Team Settings page. If you don’t have an API key yet, sign up for a free Honeycomb account.

Installation  🔗

Before you can instrument your Go application with OpenTelemetry, you first have to fetch the appropriate packages and add them to your go.mod file. Run the following command from the directory with your go.mod file:

go get go.opentelemetry.io/otel@v0.20.0 \
  go.opentelemetry.io/otel/exporters/otlp@v0.20.0 \
  go.opentelemetry.io/otel/trace@v0.20.0

Next we need to create and configure a tracer. Here we’ll use some OpenTelemetry components to describe our app and set up an exporter that will send spans to Honeycomb.

Open or create a file called main.go:

package main

import (
    "context"

    "google.golang.org/grpc/credentials"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp"
    "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/semconv"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    ctx := context.Background()

    // Create an OTLP exporter, passing in Honeycomb credentials as environment variables.
    exp, err := otlp.NewExporter(
        ctx,
        otlpgrpc.NewDriver(
            otlpgrpc.WithEndpoint("api.honeycomb.io:443"),
            otlpgrpc.WithHeaders(map[string]string{
                "x-honeycomb-team": os.Getenv("HONEYCOMB_API_KEY"),
                "x-honeycomb-dataset": os.Getenv("HONEYCOMB_DATASET"),
            }),
            otlpgrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
        )
    )

    if err != nil {
      log.Fatalf("failed to initialize exporter: %v", err)
    }

    // Create a new tracer provider with a batch span processor and the otlp exporter.
    // Add a resource attribute service.name that identifies the service in the Honeycomb UI.
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(semconv.ServiceNameKey.String("ExampleService"))),
    )

    // Handle this error in a sensible manner where possible
    defer func() { _ = tp.Shutdown(ctx) }()

    // Set the Tracer Provider and the W3C Trace Context propagator as globals
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}),
    )

    // Create a tracer instance.
    tracer := otel.Tracer("example/honeycomb-go")
}

Adding Auto-Instrumentation  🔗

The OpenTelemetry-Go Contrib repo contains extensions that provide automatic instrumentation for many popular libraries.

If you are writing a web application in Go, you can use the extensions in this repository to automatically generate and export spans from your application to Honeycomb.

In this example, we’ll focus on auto-instrumentation for servers that use the net/http package, but other popular muxers are supported.

From the directory containing your go.mod file, run the following command:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

We can now wrap our http handler functions in the net/http auto-instrumentation middleware:

import (
  // ...
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  // ...
)

// ...

handler := func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

http.Handle("/hello", otelhttp.NewHandler(http.HandlerFunc(handler), "hello"))

log.Fatal(http.ListenAndServe(":3030", nil))

Spans will now be auto-generated whenever requests are served from the handler.

Adding Context to Spans  🔗

It’s often beneficial to add context to a currently executing span in a trace. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID. This example assumes you are writing a web application with the net/http package:

import (
  // ...
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
  // ...
)

// ...
handler := func(w http.ResponseWriter, r *http.Request) {
    user := someServiceCall() // get the currently logged in user
    ctx := r.Context()
    span := trace.SpanFromContext(ctx)
    span.SetAttributes(attribute.Int("user_id", user.getID()))
}
// ...

This will add a user_id field to the current span so that you can use the field in WHERE, GROUP BY, or ORDER clauses in the Honeycomb query builder.

Creating Spans  🔗

Auto-instrumentation can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:

tracer := otel.GetTracerProvider().Tracer("") // if not already in scope
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
// ... do cool stuff

Sampling  🔗

To control how many spans are exported to Honeycomb, you can configure a sampler to send a portion of your traffic. The sampler is configured with a sample ratio between 0 (send nothing) and 1 (send everything). All spans in a sampled trace are sent to Honeycomb.

The OpenTelemetry Sampler type defines a function ShouldSample which returns a SamplingResult.

In order for Honeycomb to know the sample rate that was applied to the event being sampled, the sampler must return an attribute called SampleRate along with this result:

import (
    "go.opentelemetry.io/otel/attribute"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

var (
    ErrInvalidSampleRate = errors.New("sample rate must be >= 1")
)

type deterministicSampler struct {
        sampleRate int
}

func DeterministicSampler(sampleRate int) (*deterministicSampler, error) {
    // ...

    return &deterministicSaampler{
            sampleRate: sampleRate,
    }, nil
}

func (ds *deterministicSampler) ShouldSample(p sdktrace.SamplingParameters) sdktrace.SamplingResult {
    // ...
    return sdktrace.SamplingResult{
        Decision: sdktrace.RecordAndSample,
        Attributes: attribute.Int("SampleRate", ds.sampleRate)
    }
}

The sampler is passed as an option when creating the tracer provider:

tp := sdktrace.NewTracerProvider(
    sdktrace.WithSpanProcessor(
        sdktrace.NewBatchSpanProcessor(exp),
    ),
    sdktrace.WithResource(resource.NewWithAttributes(semconv.ServiceNameKey.String("ExampleService"))),
    sdktrace.WithSampler(DeterministicSampler(8))
)

Distributed Trace Propagation  🔗

When a service calls another service, you want to be sure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.

In order for this to work, both the sending and receiving service must be using the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.