Honeycomb has an OpenTelemetry Distribution for Go that wraps the official OpenTelemetry Go SDK. It simplifies the setup process and lets you add instrumentation to your applications and send telemetry data to Honeycomb.
These instructions will explain how to set up automatic and manual instrumentation for a service written in Go. In order to follow along, you will need:
There is an example that configures an application to send OpenTelemetry data to Honeycomb.
This section describes using instrumentation libraries with the OpenTelemetry Go SDK. For automatic instrumentation using eBPF, refer to Automatic Instrumentation using eBPF.
Install the Honeycomb OpenTelemetry Go Distribution package:
go get github.com/honeycombio/honeycomb-opentelemetry-go \
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
Prepare your application to send spans to Honeycomb.
Open or create a file called main.go
:
package main
import (
"fmt"
"log"
"net/http"
"github.com/honeycombio/honeycomb-opentelemetry-go"
"github.com/honeycombio/otel-config-go/otelconfig"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
// Implement an HTTP Handler function to be instrumented
func httpHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, World")
}
// Wrap the HTTP handler function with OTel HTTP instrumentation
func wrapHandler() {
handler := http.HandlerFunc(httpHandler)
wrappedHandler := otelhttp.NewHandler(handler, "hello")
http.Handle("/hello", wrappedHandler)
}
func main() {
// Enable multi-span attributes
bsp := honeycomb.NewBaggageSpanProcessor()
// Use the Honeycomb distro to set up the OpenTelemetry SDK
otelShutdown, err := otelconfig.ConfigureOpenTelemetry(
otelconfig.WithSpanProcessor(bsp),
)
if err != nil {
log.Fatalf("error setting up OTel SDK - %e", err)
}
defer otelShutdown()
// Initialize HTTP handler instrumentation
wrapHandler()
log.Fatal(http.ListenAndServe(":3030", nil))
}
Use environment variables to configure the Honeycomb OpenTelemetry Go distribution package:
export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"
Variable | Description |
---|---|
OTEL_SERVICE_NAME |
Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. |
HONEYCOMB_API_KEY |
Your API Key generated in Honeycomb. Learn how to find your Honeycomb API Key. |
If you are a Honeycomb Classic user, the Dataset also must be specified using the HONEYCOMB_DATASET
environment variable.
A Dataset is a bucket where data gets stored in Honeycomb.
export HONEYCOMB_DATASET="your-dataset"
This is the complete list of configuration options for the Honeycomb Distribution.
Environment Variable | Default Value | Description |
---|---|---|
HONEYCOMB_API_KEY |
None | [required – see note below] Your Honeycomb API key |
OTEL_SERVICE_NAME |
unknown_service |
[required – see note below] service.name attribute, where all trace data is sent |
HONEYCOMB_TRACES_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending traces |
HONEYCOMB_METRICS_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending metrics |
HONEYCOMB_METRICS_DATASET |
None | Honeycomb dataset where metrics will be sent |
HONEYCOMB_API_ENDPOINT |
https://api.honeycomb.io:443 |
Honeycomb ingest endpoint. |
HONEYCOMB_TRACES_API_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for traces (defaults to the value of HONEYCOMB_API_ENDPOINT ). |
HONEYCOMB_METRICS_API_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for metrics (defaults to the value of HONEYCOMB_API_ENDPOINT ). |
SAMPLE_RATE |
1 (retain all data) |
Sample rate for the deterministic sampler. Must be a positive integer. |
OTEL_METRICS_ENABLED |
false |
Enable metrics export (metrics dataset must be configured as well) |
HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS |
false |
Enable local visualizations |
DEBUG |
false |
Enable debug mode |
Run your application:
go run YOUR_APPLICATION_NAME.go
Be sure to replace YOUR_APPLICATION_NAME
with the name of your application’s main file.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
To use HTTP instead of gRPC, set the OTEL_EXPORTER_OTLP_PROTOCOL
environment variable:
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
To create spans, you need to acquire a Tracer
.
import (
// ...
"go.opentelemetry.io/otel"
// ...
)
// ...
tracer := otel.Tracer("tracer.name.here")
When you create a Tracer
, OpenTelemetry requires you to give it a name as a string.
This string is the only required parameter.
When traces are sent to Honeycomb, the name of the Tracer
is turned into the library.name
field, which can be used to show all spans created from a particular tracer.
In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.
The library.name
field is also used with traces created from instrumentation libraries.
It is often beneficial to add context to a currently executing span in a trace.
For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb.
In order to do this, get the current span from the context and set an attribute with the user ID.
This example assumes you are writing a web application with the net/http
package:
import (
// ...
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
// ...
)
// ...
handler := func(w http.ResponseWriter, r *http.Request) {
user := someServiceCall() // get the currently logged in user
ctx := r.Context()
span := trace.SpanFromContext(ctx)
span.SetAttributes(attribute.Int("user.id", user.getID()))
}
// ...
This will add a user.id
field to the current span so that you can use the field in WHERE
, GROUP BY
, or ORDER
clauses in the Honeycomb query builder.
Instrumentation libraries can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what is happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:
import (
// ...
"go.opentelemetry.io/otel"
// ...
)
// ...
tracer := otel.Tracer("my-app") // if not already in scope
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
// ...
Sometimes you will want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.
We will leverage the OpenTelemetry concept of baggage to do this.
Baggage allows you to add a key
with a value
as an attribute to every subsequent child span of the current application context, as long as you configured a BaggageSpanProcessor
when you initialized OpenTelemetry.
The example below shows how you can add a user ID attribute to multiple spans in a trace.
import (
// ...
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/otel/baggage"
// ...
)
// ...
handler := func(w http.ResponseWriter, r *http.Request) {
user := someServiceCall() // get the currently logged in user
ctx := r.Context()
// add the user ID attribute to baggage and create new context
bag := baggage.FromContext(ctx)
multiSpanAttribute, _ := baggage.NewMember("user.id", user.getID)
bag, _ = bag.SetMember(multiSpanAttribute)
ctx = baggage.ContextWithBaggage(ctx, bag)
tracer := otel.Tracer("my-app") // if not already in scope
// every subsequent span created from this context, and any of its child spans,
// will have the user ID attribute from baggage
ctx, span := tracer.Start(ctx, "expensive-operation")
defer span.End()
}
// ...
The value assigned to your key in baggage.NewMember
must be a URL-decodable string that conforms to the W3C Baggage specification.
Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.
In the OpenTelemetry Go API, there is the constant named Code
that represents the status code for a span.
The values of the Code
constant are not the same as the emitted value for a span status.
The values in this enum are only used internally in the Go SDK only.
When spans are exported, the values are converted to agree with the OpenTelemetry protobuf specification.
When spans are exported by the Go SDK, the value of Ok
is set to 1
and the value of Error
is set to 2
.
If you query this value in Honeycomb, 1
refers to Ok
and 2
refers to Error
.
This is the opposite of the in-code definition for the Go SDK.
The OpenTelemetry documentation for Go has a comprehensive set of topics on manual instrumentation.
Honeycomb’s OpenTelemetry Distribution for Go includes a sampler that can be configured to use a Honeycomb sample rate.
To configure the sampler, set the SAMPLE_RATE
environment variable:
export SAMPLE_RATE=2
If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.
When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.
Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.
Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.
Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.
By default, the Honeycomb’s OpenTelemetry Distribution for Go uses the W3C trace context format.
If you opt to use a different trace context specification than W3C, ensure that both the sending and receiving service are using the same propagation format, and that both services are configured to send data to the same Honeycomb environment.
Honeycomb’s OpenTelemetry Distribution for Go can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation.
To enable local visualizations, set the HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS
environment variable to true
:
export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true
Then, run your application:
go run main.go
The output displays the name of the root span and a link to Honeycomb that shows its trace. For example:
Trace for root-span-name
Honeycomb link: <link to Honeycomb trace>
The primary purpose of Honeycomb’s Distribution for Go is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Go, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.
The Honeycomb Distribution reads specific variables and translates them to variables understood by upstream OpenTelemetry SDK.
For example, the distribution automatically configures exporters to send telemetry data to Honeycomb’s API api.honeycomb.io
.
Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.
Follow the directions below to instrument with OpenTelemetry for Go.
go get \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
export OTEL_SERVICE_NAME="your-service-name"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
package main
import (
"context"
"fmt"
"log"
"net/http"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
func main() {
ctx := context.Background()
// Configure a new OTLP exporter using environment variables for sending data to Honeycomb over gRPC
client := otlptracegrpc.NewClient()
exp, err := otlptrace.New(ctx, client)
if err != nil {
log.Fatalf("failed to initialize exporter: %e", err)
}
// Create a new tracer provider with a batch span processor and the otlp exporter
tp := trace.NewTracerProvider(
trace.WithBatcher(exp),
)
// Handle shutdown to ensure all sub processes are closed correctly and telemetry is exported
defer func() {
_ = exp.Shutdown(ctx)
_ = tp.Shutdown(ctx)
}()
// Register the global Tracer provider
otel.SetTracerProvider(tp)
// Register the W3C trace context and baggage propagators so data is propagated across services/processes
otel.SetTextMapPropagator(
propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
),
)
// Implement an HTTP handler func to be instrumented
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, World")
})
// Setup handler instrumentation
wrappedHandler := otelhttp.NewHandler(handler, "hello")
http.Handle("/hello", wrappedHandler)
// Start web server
log.Fatal(http.ListenAndServe(":3030", nil))
}
If you are a Honeycomb Classic user, the Dataset also must be specified using the HONEYCOMB_DATASET
environment variable.
A Dataset is a bucket where data gets stored in Honeycomb.
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset"
You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.
Use a TraceIdRatioBased
sampler, with a ratio expressed as 1/N
.
Then, also create a resource attribute called SampleRate
with the value of N
.
This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.
In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).
export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2"
When using the OTEL_EXPORTER_OTLP_ENDPOINT
environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.
For example, when exporting trace data, the endpoint is updated to append v1/traces
.
When exporting metrics data, the endpoint is updated to append v1/metrics
.
The same modification is not necessary for gRPC.
By setting OTEL_EXPORTER_OTLP_ENDPOINT
to https://api.honeycomb.io
, traces are sent to https://api.honeycomb.io/v1/traces
and metrics to https://api.honeycomb.io/v1/metrics
.
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT
instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT
.
When using a signal-specific environment variable, these paths must be appended manually.
Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
for traces, appending the endpoint with v1/traces
, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
for metrics, appending the endpoint with v1/metrics
.
Send both traces and metrics to Honeycomb using this method by setting the following variables:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics
More details about endpoints and signals can be found in the OpenTelemetry Specification.
To instrument http and gRPC requests in Go typically requires wrapping requests with OpenTelemetry instrumentation libraries. There is a new project that allows for automatic instrumentation of http and gRPC requests using eBPF, which requires no application code changes.
Because the automatic instrumentation uses eBPF, it requires a Linux kernel. Automatic instrumentation should work on any Linux kernel above 4.4. For most cloud-native applications, this means including the docker image to run as an agent in a container for each application.
There are three main configuration options needed for each instrumented application:
OTEL_GO_AUTO_TARGET_EXE
, is where the agent will watch for processes.OTEL_EXPORTER_OTLP_ENDPOINT
, is where telemetry will be sent.OTEL_SERVICE_NAME
, will be used as the Service Dataset in Honeycomb, which is where data is stored.Configure an OpenTelemetry Collector to receive traces over OTLP/gRPC and export those traces to Honeycomb. Then follow the instructions for deployment in Kubernetes or from source on a Linux machine.
The automatic instrumentation agent runs in the same container node as your application.
The agent requires shareProcessNamespace
, as well as some elevated permissions in securityContext
.
Here is an example of what a deployment spec.template.spec
may look like with an existing application called “my-service”:
spec:
shareProcessNamespace: true
securityContext: {}
terminationGracePeriodSeconds: 30
containers:
- name: my-service
image: my-service:v42
ports:
- containerPort: 7007
name: http
- name: my-service-instrumentation
image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.2.0-alpha
env:
- name: OTEL_GO_AUTO_TARGET_EXE
value: /app/my-service
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector:4317
- name: OTEL_SERVICE_NAME
value: my-service-name
securityContext:
runAsUser: 0
capabilities:
add:
- SYS_PTRACE
privileged: true
There is an example that illustrates a Kubernetes deployment.
If the application is running in Linux, an alternative to Kubernetes is to build and run the instrumentation from source.
To use the instrumentation without a docker image, build a binary from source and save as otel-go-instrumentation
.
Set environment variables for the application, service name, and endpoint, and pass into a run command with the instrumentation.
Here is an example of enabling instrumentation for an application running in ~/app/my-service
:
OTEL_GO_AUTO_TARGET_EXE=~/app/my-service \ # application being instrumented
OTEL_SERVICE_NAME=my-service \ # name of service in telemetry data
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \ # send to collector
./otel-go-instrumentation
sudo
command.The service name is a required configuration value.
If it is unspecified, all trace data will be sent to a default dataset called unknown_service
.
If a This trace has multiple spans sharing the same non-null span ID
error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.
One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.
If no errors appear but your data is not in Honeycomb as expected, you can set the DEBUG
environment variable, which will both log the distribution configuration to stdout
and configure spans to be output to stdout
.
This will help confirm whether your app is being instrumented with the data you expect.
export DEBUG=true
Keep in mind that printing to the console should only be used for debugging purposes. It is not recommended for production.
By default, the Honeycomb Distribution uses a secure exporter. To export to an insecure endpoint, such as a local collector on the same network, set the Insecure option for the exporter with an environment variable:
export OTEL_EXPORTER_OTLP_INSECURE=true
To set the Insecure option in code instead of an environment variable:
otelconfig.ConfigureOpenTelemetry(otelconfig.WithExporterInsecure(true))
Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP protobuf definitions are 0.7.0 for traces and metrics.
If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.
You may receive a 464
error response from the Honeycomb API when sending telemetry using gRPC and HTTP1.
The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.