OpenTelemetry for Go | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

OpenTelemetry for Go

This guide will help you add OpenTelemetry to your Go application, show you how to add additional custom context to that instrumentation, and ensure that instrumentation data is being sent to Honeycomb.

Requirements  🔗

These instructions will explain how to set up automatic and manual instrumentation for a service written in Go. In order to follow along, you will need:

  • Go 1.14 or higher
  • An application written in Go
  • A Honeycomb API Key. You can find your API key on your Team Settings page. If you don’t have an API key yet, sign up for a free Honeycomb account.

Installation and Exporter Configuration  🔗

Before you can instrument your Go application with OpenTelemetry, you first have to fetch the appropriate packages and add them to your go.mod file. Run the following command from the directory with your go.mod file:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

Next, we’ll need to configure an exporter that will send spans to Honeycomb.

Open or create a file called main.go:

package main

import (
	"context"
	"log"
	"os"

	"google.golang.org/grpc/credentials"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
)

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	opts := []otlptracegrpc.Option{
		otlptracegrpc.WithEndpoint("api.honeycomb.io:443"),
		otlptracegrpc.WithHeaders(map[string]string{
			"x-honeycomb-team":    os.Getenv("HONEYCOMB_API_KEY"),
			"x-honeycomb-dataset": os.Getenv("HONEYCOMB_DATASET"),
		}),
		otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
	}

	client := otlptracegrpc.NewClient(opts...)
	return otlptrace.New(ctx, client)
}

func newTraceProvider(exp *otlptrace.Exporter) *sdktrace.TracerProvider {
    // The service.name attribute is required.
	resource :=
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceNameKey.String("ExampleService"),
		)

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(resource),
	)
}

func main() {
	ctx := context.Background()

	// Configure a new exporter using environment variables for sending data to Honeycomb over gRPC.
	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize exporter: %v", err)
	}

	// Create a new tracer provider with a batch span processor and the otlp exporter.
	tp := newTraceProvider(exp)

	// Handle this error in a sensible manner where possible
	defer func() { _ = tp.Shutdown(ctx) }()

	// Set the Tracer Provider and the W3C Trace Context propagator as globals
	otel.SetTracerProvider(tp)

    // Register the trace context and baggage propagators so data is propagated across services/processes.
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)
}

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, be sure not to put sensitive information in the Baggage attributes.

Adding Auto-Instrumentation  🔗

The OpenTelemetry-Go Contrib repo contains extensions that provide automatic instrumentation for many popular libraries.

If you are writing a web application in Go, you can use the extensions in this repository to automatically generate and export spans from your application to Honeycomb.

In this example, we’ll focus on auto-instrumentation for servers that use the net/http package, but other popular muxers are supported.

From the directory containing your go.mod file, run the following command:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

We can now wrap our http handler functions in the net/http auto-instrumentation middleware:

import (
  // ...
  "net/http"
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  // ...
)

// ...

// Have some http handler function to instrument
func httpHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

// Wrap the httpHandler function above with OTel instrumentation
handler := http.HandlerFunc(httpHandler)
wrappedHandler := otelhttp.NewHandler(handler, "hello")
http.Handle("/hello", wrappedHandler)

// And start the HTTP server
log.Fatal(http.ListenAndServe(":3030", nil))

Spans will now be auto-generated whenever requests are served from the handler.

Adding Attributes to Spans  🔗

It’s often beneficial to add context to a currently executing span in a trace. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID. This example assumes you are writing a web application with the net/http package:

import (
  // ...
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
  // ...
)

// ...
handler := func(w http.ResponseWriter, r *http.Request) {
    user := someServiceCall() // get the currently logged in user
    ctx := r.Context()
    span := trace.SpanFromContext(ctx)
    span.SetAttributes(attribute.Int("user.id", user.getID()))
}
// ...

This will add a user.id field to the current span so that you can use the field in WHERE, GROUP BY, or ORDER clauses in the Honeycomb query builder.

Creating Spans  🔗

Auto-instrumentation can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:

tracer := otel.GetTracerProvider().Tracer("") // if not already in scope
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
// ... do cool stuff

Sampling  🔗

func newTraceProvider(exp *otlptrace.Exporter) *sdkTrace.TracerProvider {
    resource :=
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("ExampleService"),
            attribute.Key("SampleRate").Int64(2), // additional resource attribute
        )

    return sdkTrace.NewTracerProvider(
        sdkTrace.WithBatcher(exp),
        sdkTrace.WithResource(resource),
        sdkTrace.WithSampler(sdkTrace.TraceIDRatioBased(0.5)), // sampler
    )
}

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb re-weights sampled data, so it is recommended that you set a resource attribute containing the sample rate.

In the example above, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Distributed Trace Propagation  🔗

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.

Ensure that you are setting a trace propagator like the code sample at the beginning of the page:

//...
otel.SetTextMapPropagator(
  propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{}, // this isn't strictly necessary, but it's good to set
  ),
)
//...

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.

By default, setting a trace context propagator will adhere to the W3C trace context format.

If you opt to use a different trace context specification than W3C, you’ll need to ensure that both the sending and receiving service must be using the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.

Troubleshooting  🔗

Leading or trailing newline characters are not allowed in gRPC metadata. If you accidentally add a newline character in your x-honeycomb-team or x-honeycomb-dataset values, your telemetry will not be exported, and you will see this error in your application logs when sending data over gRPC:

rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR