Instrumenting with OpenTelemetry | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Instrumenting with OpenTelemetry

Instructions  🔗

In general, any OpenTelemetry integration should carry out these two steps:

  • Set the OTLP/GRPC endpoint to: api.honeycomb.io:443
  • Add two headers to the exporter configuration:
    • x-honeycomb-team, set the value to your Honeycomb write key
    • x-honeycomb-dataset, set to the name of the target dataset

Select your application’s language for instructions on instrumenting with OpenTelemetry.

Before you can instrument your Go application with OpenTelemetry, you first have to fetch the appropriate packages and add them to your go.mod file. Run the following command from the directory with your go.mod file:

go get go.opentelemetry.io/otel@v0.20.0 \
  go.opentelemetry.io/otel/exporters/otlp@v0.20.0 \
  go.opentelemetry.io/otel/trace@v0.20.0

Next we need to create and configure a tracer. Here we’ll use some OpenTelemetry components to describe our app and set up an exporter that will send spans to Honeycomb.

Open or create a file called main.go:

package main

import (
    "context"

    "google.golang.org/grpc/credentials"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp"
    "go.opentelemetry.io/otel/exporters/otlp/otlpgrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/semconv"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    ctx := context.Background()

    // Create an OTLP exporter, passing in Honeycomb credentials as environment variables.
    exp, err := otlp.NewExporter(
        ctx,
        otlpgrpc.NewDriver(
            otlpgrpc.WithEndpoint("api.honeycomb.io:443"),
            otlpgrpc.WithHeaders(map[string]string{
                "x-honeycomb-team": os.Getenv("HONEYCOMB_API_KEY"),
                "x-honeycomb-dataset": os.Getenv("HONEYCOMB_DATASET"),
            }),
            otlpgrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
        )
    )

    if err != nil {
      log.Fatalf("failed to initialize exporter: %v", err)
    }

    // Create a new tracer provider with a batch span processor and the otlp exporter.
    // Add a resource attribute service.name that identifies the service in the Honeycomb UI.
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(semconv.ServiceNameKey.String("ExampleService"))),
    )

    // Handle this error in a sensible manner where possible
    defer func() { _ = tp.Shutdown(ctx) }()

    // Set the Tracer Provider and the W3C Trace Context propagator as globals
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}),
    )

    // Create a tracer instance.
    tracer := otel.Tracer("example/honeycomb-go")
}

Adding Auto-Instrumentation  🔗

The OpenTelemetry-Go Contrib repo contains extensions that provide automatic instrumentation for many popular libraries.

If you are writing a web application in Go, you can use the extensions in this repository to automatically generate and export spans from your application to Honeycomb.

In this example, we’ll focus on auto-instrumentation for servers that use the net/http package, but other popular muxers are supported.

From the directory containing your go.mod file, run the following command:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

We can now wrap our http handler functions in the net/http auto-instrumentation middleware:

import (
  // ...
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  // ...
)

// ...

handler := func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

http.Handle("/hello", otelhttp.NewHandler(http.HandlerFunc(handler), "hello"))

log.Fatal(http.ListenAndServe(":3030", nil))

Spans will now be auto-generated whenever requests are served from the handler.

To instrument your Python app with OpenTelemetry, you will need to install these packages:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp-proto-grpc

Next, we need to create and configure a tracer instance. Here we’ll use some OpenTelemetry components to describe our app and set up an exporter that will send spans to Honeycomb.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from grpc import ssl_channel_credentials

# resource describes app-level information that will be added to all spans
resource = Resource(attributes={
    "service.name": os.environ.get("SERVICE_NAME")
})

# create new trace provider with our resource
trace_provider = TracerProvider(resource=resource)

# create exporter to send spans to Honeycomb
otlp_exporter = OTLPSpanExporter(
    endpoint="api.honeycomb.io:443",
    insecure=False,
    credentials=ssl_channel_credentials(),
    headers=(
        ("x-honeycomb-team", os.environ.get("HONEYCOMB_API_KEY")),
        ("x-honeycomb-dataset", os.environ.get("HONEYCOMB_DATASET"))
    )
)

# register exporter with provider
trace_provider.add_span_processor(
    BatchSpanProcessor(otlp_exporter)
)

# register trace provider
trace.set_tracer_provider(trace_provider)

Note that we’re setting a few configuration options using environment variables. You can set these as inline environment variables like this when starting your python app:

HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_DATASET=my-dataset \
SERVICE_NAME=my-favorite-service \
python app.py

Install the Honeycomb.OpenTelemetry nuget package using a NuGet package manager or the dotnet CLI.

For example, with the dotnet CLI, use:

dotnet add package Honeycomb.OpenTelemetry

Configure your app for Honeycomb OpenTelemetry Distribution for .NET with one of the following configuration methods:

Programmatically  🔗

You can provide an HoneycombOptions object to programmatically configure the OpenTelemetry SDK.

var tracerProvider = new TracerProviderBuilder()
    .UseHoneycomb(new HoneycombOptions
    {
        ServiceName = "my-app",
        ApiKey = "{apikey}",
        Dataset = "{dataset}"
    })
    .Build();

ASP.NET Core  🔗

You can provide an appsettings.json file:

{
  ...
  "Honeycomb": {
    "ServiceName": "my-app",
    "ApiKey": "{apikey}",
    "Dataset": "{dataset}",
  }
}

Then, pass the IConfiguration object to configure the OpenTelemetry SDK:

var tracerProvider = new TracerProviderBuilder()
    .UseHoneycomb(configuration)
    .Build();

NOTE: It is possible to leverage the AppSettings ability to read environment variables to populate values in the configuration object. For example, the following environment variables could be used to override the values in an AppSettings file:

export HONEYCOMB__SERVICENAME=my-app
export HONEYCOMB__APIKEY={apikey}
export HONEYCOMB__DATASET={dataset}

See the OpenTelemetry ASP.NET Core instrumentation for more advanced options.

ASP.NET (.NET Framework v4.6.1+)  🔗

Add the following to your Web.config:

<system.webServer>
    <modules>
    <add name="TelemetryCorrelationHttpModule"
         type="Microsoft.AspNet.TelemetryCorrelation.TelemetryCorrelationHttpModule,Microsoft.AspNet.TelemetryCorrelation"
         preCondition="integratedMode,managedHandler" />
    </modules>
</system.webServer>

Configure the OpenTelemetry SDK to use Honeycomb during application start:

using OpenTelemetry;
using OpenTelemetry.Trace;
using Honeycomb.OpenTelemetry;

public class WebApiApplication : HttpApplication
{
    private TracerProvider tracerProvider;

    protected void Application_Start()
    {
        this.tracerProvider = new TracerProviderBuilder()
            .UseHoneycomb(new HoneycombOptions
            {
                ServiceName = "my-app",
                ApiKey = "{apikey}",
                Dataset = "{dataset}"
            })
            .Build();
    }

    protected void Application_End()
    {
        this.tracerProvider?.Dispose();
    }
}

See the OpenTelemetry ASP.NET instrumentation for more advanced options.

Command Line Arguments  🔗

You can provide command line arguments when starting your application:

dotnet run \
  --service-name=my-app \
  --honeycomb-apikey={apikey} \
  --honeycomb-dataset={dataset}

Then, pass the command line arguments to configure the OpenTelemetry SDK:

var tracerProvider = new TracerProviderBuilder()
    .UseHoneycomb(args)
    .Build();

NOTE: UseHoneycomb sets the OpenTelemetry SDK ResourceBuilder, which is used to create the Resource that is associated to all spans created by the tracer. Please do not override the ResourceBuilder by calling SetResourceBuilder with another instance. Otherwise, some resource attributes may not be queryable in your Honeycomb data.

The auto-instrumentation agent for Honeycomb OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is specified as a command-line argument when invoking the Java Virtual Machine (JVM).

In order to use the auto-instrumentation agent, you must first download and distribute the JAR file.

The agent JAR file must be available when running the java process that starts your service. The file can be downloaded from the GitHub Releases page or you can download it with a CLI tool like curl:

curl -LO https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/download/v0.4.0/honeycomb-opentelemetry-javaagent-0.4.0-all.jar

Once the JAR file is downloaded, you can configure the agent with Honeycomb credentials.

Java Agent Configuration  🔗

The Honeycomb Agent has the following configuration options, which can be specified as system properties or environment variables. System properties take precedence over environment variables if both are specified.

System Property Environment Variable Description
honeycomb.api.key HONEYCOMB_API_KEY [required] Your Honeycomb API key
honeycomb.dataset HONEYCOMB_DATASET [required] Honeycomb dataset where traces will be sent
honeycomb.api.endpoint HONEYCOMB_API_ENDPOINT [optional] Honeycomb ingest endpoint (defaults to https://api.honeycomb.io:443).
sample.rate SAMPLE_RATE [optional] Sample rate for the deterministic sampler (defaults to 1, always sample)
service.name SERVICE_NAME [optional] service.name attribute to be used for all spans (defaults to empty)

These configuration options are specified when invoking the JVM. The following examples assume that the JAR file is in the current working directory. Configure the auto-instrumentation agent using environment variables:

SAMPLE_RATE=2 \
SERVICE_NAME=my-favorite-service \
HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_DATASET=my-dataset \
java -javaagent:honeycomb-opentelemetry-javaagent-0.4.0-all.jar -jar /path/to/myapp.jar

Or, configure using system properties:

java \
-Dsample.rate=2 \
-Dservice.name=my-favorite-service \
-Dhoneycomb.api.key=my-api-key \
-Dhoneycomb.dataset=my-dataset \
-javaagent:honeycomb-opentelemetry-javaagent-0.4.0-all.jar -jar /path/to/myapp.jar

To auto-instrument your Node.js Express application, you need to add the following packages:

npm install --save @grpc/grpc-js
npm install --save @opentelemetry/api
npm install --save @opentelemetry/sdk-node
npm install --save @opentelemetry/exporter-collector-grpc
npm install --save @opentelemetry/auto-instrumentations-node

auto-instrumentations-node is a meta package that provides a simple way to initialize multiple Node.js instrumentations. Alternatively, you can install individual instrumentations.

Initialization  🔗

The OpenTelemetry initialization needs to happen first in your application lifecycle. You can accomplish this by creating an initialization file. You can then import the file as the first step in you application initialization, or include it with the -r, --require Node.js CLI option.

OpenTelemetry initialization:

// tracing.js

const process = require('process');
const { Metadata, credentials } = require("@grpc/grpc-js");

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { CollectorTraceExporter } = require("@opentelemetry/exporter-collector-grpc");

const metadata = new Metadata()
metadata.set('x-honeycomb-team', '<YOUR_API_KEY>');
metadata.set('x-honeycomb-dataset', '<YOUR_DATASET>');
const traceExporter = new CollectorTraceExporter({
  url: 'grpc://api.honeycomb.io:443/',
  credentials: credentials.createSsl(),
  metadata
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: '<YOUR_SERVICE_NAME>',
  }),
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start()
  .then(() => console.log('Tracing initialized'))
  .catch((error) => console.log('Error initializing tracing', error));

process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Running the Node.js app with OpenTelemetry initialization file:

node -r ./tracing.js app.js

You can see a complete example here.

Follow the instructions below to set up the OpenTelemetry OTLP Exporter for Ruby.

Installation  🔗

To auto-instrument your Ruby application, add these gems to your Gemfile:

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-instrumentation-all'

The inclusion of opentelemetry-instrumentation-all in the above list provides instrumentations for Rails, Sinatra, several HTTP libraries, and more.

Ruby Initialization  🔗

The OpenTelemetry initialization needs to happen early in your application lifecycle. For Rails applications, the usual way to initialize OpenTelemetry is in a Rails initializer. For other Ruby services, perform this initialization as early as possible in the start-up process.

OpenTelemetry initialization:

# config/initializers/opentelemetry.rb
require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
require 'opentelemetry/instrumentation/all'
OpenTelemetry::SDK.configure do |c|
  c.service_name = '<YOUR_SERVICE_NAME>'
  c.use_all() # enables all instrumentation!
end

A complete example can be viewed here.

Choose the API and SDK for your language, or search the OpenTelemetry Registry for the term “instrumentation”.

Follow the instructions to install the required packages. Configure OpenTelemetry in your service as documented in the language’s OpenTelemetry SDK for an otlp GRPC exporter.

Or gather data from multiple services and control data egress, by exporting to an OpenTelemetry Collector on your network configured to forward onto Honeycomb.

Resources  🔗

Table of Contents