OpenTelemetry for Go | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

OpenTelemetry for Go

This guide will help you add OpenTelemetry to your Go application, show you how to add additional custom context to that instrumentation, and ensure that instrumentation data is being sent to Honeycomb.

Requirements 

These instructions will explain how to set up automatic and manual instrumentation for a service written in Go. In order to follow along, you will need:

  • Go 1.14 or higher
  • An application written in Go
  • A Honeycomb API Key. You can find your API key on your Team Settings page. If you do not have an API key yet, sign up for a free Honeycomb account.

Adding Auto-Instrumentation 

Acquire Dependencies 

Fetch OpenTelemetry packages so you can configure an HTTP server to automatically generate trace data:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Initialize 

Configure an exporter to send spans to Honeycomb.

Open or create a file called main.go:

package main

import (
    "context"
    "fmt"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"

    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
    client := otlptracehttp.NewClient()
    return otlptrace.New(ctx, client)
}

func newTraceProvider(exp *otlptrace.Exporter) *sdktrace.TracerProvider {
    return sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
    )
}

// Implement an HTTP Handler func to be instrumented
func httpHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

// Wrap the HTTP handler func with OTel HTTP instrumentation
func wrapHandler() {
    handler := http.HandlerFunc(httpHandler)
    wrappedHandler := otelhttp.NewHandler(handler, "hello")
    http.Handle("/hello", wrappedHandler)
}

func main() {
    ctx := context.Background()

    // Configure a new exporter using environment variables for sending data to Honeycomb over gRPC.
    exp, err := newExporter(ctx)
    if err != nil {
        log.Fatalf("failed to initialize exporter: %v", err)
    }

    // Create a new tracer provider with a batch span processor and the otlp exporter.
    tp := newTraceProvider(exp)

    // Handle this error in a sensible manner where possible
    defer func() { _ = tp.Shutdown(ctx) }()

    // Set the Tracer Provider and the W3C Trace Context propagator as globals
    otel.SetTracerProvider(tp)

    // Register the trace context and baggage propagators so data is propagated across services/processes.
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    )

    // Initialize HTTP handler instrumentation
    wrapHandler()
	log.Fatal(http.ListenAndServe(":3030", nil))
}

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, be sure not to put sensitive information in the Baggage attributes.

Configure and Run 

If using the dataset-only data model, refer to the Honeycomb Classic tab for instructions. Not sure? Learn more about Honeycomb versus Honeycomb Classic.

Configure OpenTelemetry to send events to Honeycomb using environment variables.

The header x-honeycomb-team is your API key. The header x-honeycomb-dataset is the name of a Dataset that you need to specify. A Dataset is a bucket where data gets stored in Honeycomb.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset"
export OTEL_SERVICE_NAME="your-service-name"

Now you can run your application and see traces created for each incoming HTTP request:

go run main.go

Configure OpenTelemetry to send events to Honeycomb using environment variables.

The header x-honeycomb-team is your API key. Your service name will be used as the Service Dataset in Honeycomb, which is where data is stored. The service name is specified by OTEL_SERVICE_NAME.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
export OTEL_SERVICE_NAME="your-service-name"

Now you can run your application and see traces created for each incoming HTTP request:

go run main.go

gRPC Instead of HTTP 

To use gRPC instead of HTTP, install the package, replace the import, adjust the exporter, and update the endpoint:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

...

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
    client := otlptracegrpc.NewClient()
    return otlptrace.New(ctx, client)
}
export OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443

Adding Attributes to Spans 

It is often beneficial to add context to a currently executing span in a trace. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID. This example assumes you are writing a web application with the net/http package:

import (
  // ...
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/trace"
  // ...
)

// ...
handler := func(w http.ResponseWriter, r *http.Request) {
    user := someServiceCall() // get the currently logged in user
    ctx := r.Context()
    span := trace.SpanFromContext(ctx)
    span.SetAttributes(attribute.Int("user.id", user.getID()))
}
// ...

This will add a user.id field to the current span so that you can use the field in WHERE, GROUP BY, or ORDER clauses in the Honeycomb query builder.

Creating Spans 

Auto-instrumentation can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what is happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:

tracer := otel.GetTracerProvider().Tracer("") // if not already in scope
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
// ... do cool stuff

Sampling 

func newTraceProvider(exp *otlptrace.Exporter) *sdkTrace.TracerProvider {
    r, err :=
        resource.Merge(
            resource.Default(),
            resource.NewWithAttributes(
                semconv.SchemaURL,
                semconv.ServiceNameKey.String("ExampleService"),
                attribute.Key("SampleRate").Int64(2), // additional resource attribute
            )
    )

    if err != nil {
        panic(err)
    }

    return sdkTrace.NewTracerProvider(
        sdkTrace.WithBatcher(exp),
        sdkTrace.WithResource(r),
        sdkTrace.WithSampler(sdkTrace.TraceIDRatioBased(0.5)), // sampler
    )
}

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb re-weights sampled data, so it is recommended that you set a resource attribute containing the sample rate.

In the example above, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.

Ensure that you are setting a trace propagator like the code sample at the beginning of the page:

//...
otel.SetTextMapPropagator(
  propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{}, // this is not strictly necessary, but it is good to set
  ),
)
//...

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.

By default, setting a trace context propagator will adhere to the W3C trace context format.

If you opt to use a different trace context specification than W3C, you will need to ensure that both the sending and receiving service must be using the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.

Endpoint URLs for OTLP/HTTP 

When using the OTEL_EXPORTER_OTLP_ENDPOINT environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.

For example, when exporting trace data, the endpoint is updated to append v1/traces. When exporting metrics data, the endpoint is updated to append v1/metrics. The same modification is not necessary for gRPC.

By setting OTEL_EXPORTER_OTLP_ENDPOINT to https://api.honeycomb.io, traces are sent to https://api.honeycomb.io/v1/traces and metrics to https://api.honeycomb.io/v1/metrics.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io

If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT.

When using a signal-specific environment variable, these paths must be appended manually. Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces, appending the endpoint with v1/traces, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT for metrics, appending the endpoint with v1/metrics.

Send both traces and metrics to Honeycomb using this method by setting the following variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics

More details about endpoints and signals can be found in the OpenTelemetry Specification.

Troubleshooting 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

Printing to the Console 

If no errors appear but your data is not in Honeycomb as expected, you can use a console exporter to print your spans to the console. This will help confirm whether your app is being instrumented with the data you expect.

Import stdouttrace, and temporarily replace the OTLP Exporter and TracerProvider:

import (
  // ...
  "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
  // ...
)

func initTracer() {
  var err error
  exp, err := stdouttrace.New(stdouttrace.WithPrettyPrint()) // for debugging
  if err != nil {
    log.Panicf("failed to initialize stdouttrace exporter %v\n", err)
    return
  }

  ssp := sdktrace.NewSimpleSpanProcessor(exp)
  tp = sdktrace.NewTracerProvider(
    sdktrace.WithSpanProcessor(ssp),
  )
  otel.SetTracerProvider(tp)
}

Using a logging package like stdr allows you to adjust the logging level for debugging, and to see SDK status messages in the console. The higher the number used in SetVerbosity, the more information will be emitted. Conversely, the lower the number, the less verbose the information will be. A more detailed example can be found in the OpenTelemetry for Go GitHub repository.

Import stdr and add before initializing your tracer in your main application, adjusting the logging level as needed:

import (
  // ...
  "github.com/go-logr/stdr"
  // ...
)

func main() {
  stdr.SetVerbosity(5) // set for debugging
  // then initialize tracer
  initTracer()
...
}

Keep in mind that printing to the console should only be used for debugging purposes. It is not recommended for production.

OTLP Protobuf Definitions 

Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC and HTTP/protobuf. Currently supported versions of OTLP protobuf definitions are 0.7.0 through 0.16.0 for traces, and 0.7.0 through 0.11.0 for metrics.

If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is newer than this range, temporarily use an SDK with a supported version until the newer version is supported. If the SDK’s protobuf version is older than this range, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.

Receiving 464 Errors 

You may receive a 464 error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.

Did you find what you were looking for?