> ## Documentation Index > Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt > Use this file to discover all available pages before exploring further. # Send Data with the OpenTelemetry Go SDK > Instrument your Go application with the OpenTelemetry Go SDK and send traces and metrics to Honeycomb. Use the OpenTelemetry Go SDK to instrument Go applications in a standard, vendor-agnostic, and future-proof way and send telemetry data to Honeycomb. In this guide, we will walk you through instrumenting with OpenTelemetry for Go, which will include adding automatic instrumentation to your application. ## Before You Begin Before you can set up automatic instrumentation for your Go application, you will need to do a few things. ### Prepare Your Development Environment To complete the required steps, you will need: * A working Go environment * An application written in Go ### Get Your Honeycomb API Key To send data to Honeycomb, you'll need to [sign up for a free Honeycomb account](https://ui.honeycomb.io/signup) and [create a Honeycomb Ingest API Key](/configure/environments/manage-api-keys/#create-api-key). To get started, you can create a key that you expect to swap out when you deploy to production. Name it something helpful, perhaps noting that it's a getting started key. Make note of your API key; for security reasons, you will not be able to see the key again, and you will need it later! For setup, make sure you check the "Can create datasets" checkbox so that your data will show up in Honeycomb. Later, when you replace this key with a permanent one, you can uncheck that box. If you want to use an API key you previously stored in a secure location, you can also [look up details for Honeycomb API Keys](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings, and use them to retrieve keys from your storage location. ## Configure OpenTelemetry SDK To configure the OpenTelemetry SDK and enable automatic instrumentation of HTTP requests in your application, you will add the following packages to your application. ### Acquire Dependencies Install OpenTelemetry Go packages: ```shell theme={} go get \ go.opentelemetry.io/contrib/otelconf \ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp ``` ### Initialize Prepare your application to send spans to Honeycomb. Open or create a file called `main.go`: ```go theme={} package main import ( "context" "fmt" "log" "net/http" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "go.opentelemetry.io/contrib/otelconf" "go.opentelemetry.io/otel" ) // Implement an HTTP Handler function to be instrumented func httpHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, World") } func main() { // use otelconf to set up OpenTelemetry SDK sdk, err := otelconf.NewSDK() if err != nil { log.Fatalf("error setting up OTel SDK - %e", err) } defer sdk.Shutdown(context.Background()) otel.SetTracerProvider(sdk.TracerProvider()) otel.SetTextMapPropagator(sdk.Propagator()) // Initialize HTTP handler instrumentation handler := http.HandlerFunc(httpHandler) wrappedHandler := otelhttp.NewHandler(handler, "hello") http.Handle("/hello", wrappedHandler) // Serve HTTP server log.Fatal(http.ListenAndServe(":3030", nil)) } ``` ### Configure Environment Variables Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # (*) readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: ${HONEYCOMB_METRICS_DATASET} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Fields marked with an asterisk (\*) are required for exporting metrics to Honeycomb. Set the following environment variables before running your application: | Environment Variable | Value | | :-------------------------- | :-------------------------------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | | `HONEYCOMB_METRICS_DATASET` | The Honeycomb dataset to send metrics to (\*) | If you are using the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` in the configuration file. You can also configure the SDK using [environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/general/). If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must specify the Dataset using the `HONEYCOMB_DATASET` environment variable. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` ### Run Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml go run YOUR_APPLICATION_NAME.go ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. ## Add Custom Instrumentation Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add custom, or manual, instrumentation where appropriate. Follow the instructions below to add custom instrumentation to your code. To learn more about custom, or manual, instrumentation, visit the comprehensive set of topics covered by [Manual Instrumentation for Go](https://opentelemetry.io/docs/languages/go/instrumentation/) in OpenTelemetry's documentation. ### Acquire a Tracer To create spans, you need to acquire a `Tracer`. ```go theme={} import ( // ... "go.opentelemetry.io/otel" // ... ) // ... tracer := otel.Tracer("tracer.name.here") ``` When you create a `Tracer`, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the `Tracer` is turned into the `library.name` field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different "layers" of your application, then use the name that corresponds to that "layer". The `library.name` field is also used with traces created from instrumentation libraries. ### Add Attributes to Spans Adding context to a currently executing span in a trace can be useful. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. To do this, get the current span from the context and set an attribute with the user ID. This example assumes you are writing a web application with the `net/http` package: ```go theme={} import ( // ... "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" // ... ) // ... handler := func(w http.ResponseWriter, r *http.Request) { user := someServiceCall() // get the currently logged in user ctx := r.Context() span := trace.SpanFromContext(ctx) span.SetAttributes(attribute.Int("user.id", user.getID())) } // ... ``` This will add a `user.id` field to the current span, so you can use the field in `WHERE`, `GROUP BY`, or `ORDER` clauses in the Honeycomb query builder. ### Create Spans To get the full picture of what is happening, you can leverage manual instrumentation to create custom spans that describe what is happening in your application. To do this, grab the tracer from the OpenTelemetry API: ```go theme={} import ( // ... "go.opentelemetry.io/otel" // ... ) // ... tracer := otel.Tracer("my-app") // if not already in scope ctx, span := tracer.Start(ctx, "expensive-operation") defer span.End() // ... ``` ### Add Multi-Span Attributes Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute to multiple spans, leverage the OpenTelemetry concept of [baggage](https://opentelemetry.io/docs/concepts/signals/baggage/). Baggage allows you to add a `key` with a `value` as an attribute to every subsequent child span of the current application context, as long as you configured a `BaggageSpanProcessor` when you [initialized OpenTelemetry](#configure-opentelemetry-sdk). 1. Install the `baggagetrace` package in your terminal: ```shell theme={} go get go.opentelemetry.io/contrib/processors/baggage/baggagetrace ``` 2. When configuring the OpenTelemetry SDK tracer provider, add the baggage span processor: ```golang theme={} import ( // ... "go.opentelemetry.io/contrib/processors/baggage/baggagetrace" // ... ) // Create a new tracer provider with the baggage span processor tp := trace.NewTracerProvider( baggagetrace.New() // ... ) ``` 3. Add a baggage entry for the current trace and replace `key` and `value` with your desired key-value pair: ```go theme={} import ( // ... "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" "go.opentelemetry.io/otel/baggage" // ... ) // ... handler := func(w http.ResponseWriter, r *http.Request) { ctx := r.Context() // add the user ID attribute to baggage and create new context bag := baggage.FromContext(ctx) multiSpanAttribute, _ := baggage.NewMember("key", "value") bag, _ = bag.SetMember(multiSpanAttribute) ctx = baggage.ContextWithBaggage(ctx, bag) tracer := otel.Tracer("my-app") // if not already in scope // every subsequent span created from this context, and any of its child spans, // will have the user ID attribute from baggage ctx, span := tracer.Start(ctx, "expensive-operation") defer span.End() } // ... ``` Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do **NOT** put sensitive information in the Baggage attributes. ## Automatic Instrumentation using eBPF To instrument http and gRPC requests in Go, usually you must wrap requests with OpenTelemetry instrumentation libraries. However, a [new project](https://github.com/open-telemetry/opentelemetry-go-instrumentation) allows for automatic instrumentation of http and gRPC requests using [eBPF](/get-started/basics/observability/concepts/ebpf/), which requires no application code changes. Because the automatic instrumentation uses eBPF, it requires a Linux kernel. Automatic instrumentation should work on any Linux kernel above 4.4. For most cloud-native applications, this means you must include the Docker image to run as an agent in a container for each application. ### Configure You must configure the following options for each instrumented application: * Your application path, specified by `OTEL_GO_AUTO_TARGET_EXE`, is where the agent will watch for processes. * Your endpoint, specified by `OTEL_EXPORTER_OTLP_ENDPOINT`, is where telemetry will be sent. * Your service name, specified by `OTEL_SERVICE_NAME`, will be used as the Service Dataset in Honeycomb, which is where data is stored. Configure an [OpenTelemetry Collector](/send-data/opentelemetry/collector/) to receive traces over OTLP/gRPC and export those traces to Honeycomb. Then follow the instructions for [deployment in Kubernetes](#running-in-kubernetes) or [from source on a Linux machine](#running-on-a-linux-machine). ### Running in Kubernetes The automatic instrumentation agent runs in the same container node as your application. The agent requires `shareProcessNamespace`, as well as some elevated permissions in `securityContext`. The following example shows what a deployment `spec.template.spec` could look like with an existing application called "my-service": ```yaml theme={} spec: shareProcessNamespace: true securityContext: {} terminationGracePeriodSeconds: 30 containers: - name: my-service image: my-service:v42 ports: - containerPort: 7007 name: http - name: my-service-instrumentation image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.2.0-alpha env: - name: OTEL_GO_AUTO_TARGET_EXE value: /app/my-service - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://otel-collector:4317 - name: OTEL_SERVICE_NAME value: my-service-name securityContext: runAsUser: 0 capabilities: add: - SYS_PTRACE privileged: true ``` #### Example If you prefer to learn by example, we provide an [example application](https://github.com/honeycombio/example-greeting-service/tree/main/go-auto-instrumented) that illustrates a Kubernetes deployment. ### Running on a Linux Machine If the application is running in Linux, an alternative to Kubernetes is to build and run the instrumentation from source. To use the instrumentation without a Docker image, build a binary [from source](https://github.com/open-telemetry/opentelemetry-go-instrumentation) and save as `otel-go-instrumentation`. Set environment variables for the application, service name, and endpoint, and pass into a run command with the instrumentation. The following example shows how to enable instrumentation for an application running in `~/app/my-service`: ```bash theme={} OTEL_GO_AUTO_TARGET_EXE=~/app/my-service \ # application being instrumented OTEL_SERVICE_NAME=my-service \ # name of service in telemetry data OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \ # send to collector ./otel-go-instrumentation ``` Because eBPF has powerful capabilities, running this instrumentation may require additional privileges on the host, such as running with the `sudo` command. ## Sampling You can configure the OpenTelemetry SDK to [sample the data](/manage-data-volume/sample/guidelines/) it generates. Honeycomb [weights sampled data based on sample rate](/manage-data-volume/sample/sampled-data-in-honeycomb/), so you must set a resource attribute containing the sample rate. Use a [`TraceIdRatioBased` sampler](https://opentelemetry.io//docs/specs/otel/trace/sdk/#traceidratiobased), with a ratio expressed as `1/N`. Then, also create a resource attribute called `SampleRate` with the value of `N`. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data. In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5). ```shell theme={} export OTEL_TRACES_SAMPLER="traceidratio" export OTEL_TRACES_SAMPLER_ARG=0.5 export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2" ``` ## Choosing between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. ## Endpoint URLs for OTLP/HTTP When using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable with an SDK and an HTTP exporter, the final path of the endpoint is modified by the SDK to represent the specific signal being sent. For example, when exporting trace data, the endpoint is updated to append `v1/traces`. When exporting metrics data, the endpoint is updated to append `v1/metrics`. So, if you were to set the `OTEL_EXPORTER_OTLP_ENDPOINT` to `https://api.honeycomb.io`, traces would be sent to `https://api.honeycomb.io/v1/traces` and metrics would be sent to `https://api.honeycomb.io/v1/metrics`. The same modification is not necessary for gRPC. ```shell theme={} export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io # EU instance ``` If the desired outcome is to send data to a different endpoint depending on the signal, use `OTEL_EXPORTER_OTLP__ENDPOINT` instead of the more generic `OTEL_EXPORTER_OTLP_ENDPOINT`. When using a signal-specific environment variable, these paths must be appended manually. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` for traces, appending the endpoint with `v1/traces`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` for metrics, appending the endpoint with `v1/metrics`. Send both traces and metrics to Honeycomb using this method by setting the following variables: ```shell theme={} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces # US instance #export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.eu1.honeycomb.io/v1/traces # EU instance export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics # US instance #export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.eu1.honeycomb.io/v1/metrics # EU instance ``` More details about endpoints and signals can be found in the [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md). ## Troubleshooting To explore common issues when sending data, visit [Common Issues with Sending Data to Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). ### Errors Sending to Honeycomb If OpenTelemetry is unable to send spans to Honeycomb, it should print an error to the console. For example, this error indicates that the Honeycomb API key is missing or incorrect: ```sh theme={} 2025/02/20 15:56:34 traces export: failed to send to https://api.honeycomb.io/v1/traces: 401 Unauthorized ``` #### Nothing Sent and No Errors If no events arrive in Honeycomb and no errors are printed, try increasing the logging verbosity of the OpenTelemetry SDK before initialization: ```go theme={} import( "log" "os" "github.com/go-logr/stdr" ) stdr.SetVerbosity(100) logger := stdr.New(log.New(os.Stdout, "otel-logger ", log.LstdFlags|log.Lshortfile)) otel.SetLogger(logger) ``` This results in log statements like the following: ```sh theme={} otel-logger 2025/02/20 16:02:26 internal_logging.go:45: "level"=4 "msg"="TracerProvider created" "config"={"SpanProcessors"=[{"Type"="BatchSpanProcessor" "SpanExporter"={"Type"="otlptrace" "Client"={"Type"="otlphttphttp" "Endpoint"="api.honeycomb.io" "Insecure"=false}} "Config"={"MaxQueueSize"=2048 "BatchTimeout"="5s" "ExportTimeout"="30s" "MaxExportBatchSize"=512 "BlockOnQueueFull"=false}}] "SamplerType"="trace.parentBased" "IDGeneratorType"="*trace.randomIDGenerator" "SpanLimits"={"AttributeValueLengthLimit"=-1 "AttributeCountLimit"=128 "EventCountLimit"=128 "LinkCountLimit"=128 "AttributePerEventCountLimit"=128 "AttributePerLinkCountLimit"=128} "Resource"={"Attributes"={"telemetry.sdk.version"="1.26.0" "env.whereami"="somewhere" "service.name"="awesome-sauce" "telemetry.sdk.language"="go" "telemetry.sdk.name"="opentelemetry"} "SchemaURL"="https://opentelemetry.io/schemas/1.24.0"}} ``` A crucial piece of this log says that OpenTelemetry is configured to send to Honeycomb: ```sh theme={} "SpanExporter"={"Type"="otlptrace" "Client"={"Type"="otlphttphttp" "Endpoint"="api.honeycomb.io" "Insecure"=false}} ``` If the `SpanExporter` is missing, there is a problem with your exporter configuration. If it looks right, try [sending a custom span](#create-spans) immediately after initialization as a further test.