Instrumenting with OpenTelemetry | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Instrumenting with OpenTelemetry

Instructions  🔗

In general, any OpenTelemetry integration should carry out these two steps:

  • Set the endpoint:
    • If using OTLP/gRPC, set it to: api.honeycomb.io:443
    • If using OTLP/HTTPS, set it to: https://api.honeycomb.io:443
  • Add two headers to the exporter configuration:
    • x-honeycomb-team, set the value to your Honeycomb write key
    • x-honeycomb-dataset, set to the name of the target dataset

Select your application’s language for instructions on instrumenting with OpenTelemetry.

Before you can instrument your Go application with OpenTelemetry, you first have to fetch the appropriate packages and add them to your go.mod file. Run the following command from the directory with your go.mod file:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

Next, we’ll need to configure an exporter that will send spans to Honeycomb.

Open or create a file called main.go:

package main

import (
	"context"
	"log"
	"os"

	"google.golang.org/grpc/credentials"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
)

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	opts := []otlptracegrpc.Option{
		otlptracegrpc.WithEndpoint("api.honeycomb.io:443"),
		otlptracegrpc.WithHeaders(map[string]string{
			"x-honeycomb-team":    os.Getenv("HONEYCOMB_API_KEY"),
			"x-honeycomb-dataset": os.Getenv("HONEYCOMB_DATASET"),
		}),
		otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
	}

	client := otlptracegrpc.NewClient(opts...)
	return otlptrace.New(ctx, client)
}

func newTraceProvider(exp *otlptrace.Exporter) *sdktrace.TracerProvider {
    // The service.name attribute is required.
	resource :=
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceNameKey.String("ExampleService"),
		)

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(resource),
	)
}

func main() {
	ctx := context.Background()

	// Configure a new exporter using environment variables for sending data to Honeycomb over gRPC.
	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize exporter: %v", err)
	}

	// Create a new tracer provider with a batch span processor and the otlp exporter.
	tp := newTraceProvider(exp)

	// Handle this error in a sensible manner where possible
	defer func() { _ = tp.Shutdown(ctx) }()

	// Set the Tracer Provider and the W3C Trace Context propagator as globals
	otel.SetTracerProvider(tp)

    // Register the trace context and baggage propagators so data is propagated across services/processes.
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)
}

Adding Auto-Instrumentation  🔗

The OpenTelemetry-Go Contrib repo contains extensions that provide automatic instrumentation for many popular libraries.

If you are writing a web application in Go, you can use the extensions in this repository to automatically generate and export spans from your application to Honeycomb.

In this example, we’ll focus on auto-instrumentation for servers that use the net/http package, but other popular muxers are supported.

From the directory containing your go.mod file, run the following command:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

We can now wrap our http handler functions in the net/http auto-instrumentation middleware:

import (
  // ...
  "net/http"
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  // ...
)

// ...

// Have some http handler function to instrument
func httpHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World")
}

// Wrap the httpHandler function above with OTel instrumentation
handler := http.HandlerFunc(httpHandler)
wrappedHandler := otelhttp.NewHandler(handler, "hello")
http.Handle("/hello", wrappedHandler)

// And start the HTTP server
log.Fatal(http.ListenAndServe(":3030", nil))

Spans will now be auto-generated whenever requests are served from the handler.

To instrument your Python app with OpenTelemetry, you will need to install these packages:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp-proto-grpc

Next, we need to create and configure a tracer instance. Here we’ll use some OpenTelemetry components to describe our app and set up an exporter that will send spans to Honeycomb.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from grpc import ssl_channel_credentials

# resource describes app-level information that will be added to all spans
resource = Resource(attributes={
    "service.name": os.environ.get("SERVICE_NAME")
})

# create new trace provider with our resource
trace_provider = TracerProvider(resource=resource)

# create exporter to send spans to Honeycomb
otlp_exporter = OTLPSpanExporter(
    endpoint="api.honeycomb.io:443",
    insecure=False,
    credentials=ssl_channel_credentials(),
    headers=(
        ("x-honeycomb-team", os.environ.get("HONEYCOMB_API_KEY")),
        ("x-honeycomb-dataset", os.environ.get("HONEYCOMB_DATASET"))
    )
)

# register exporter with provider
trace_provider.add_span_processor(
    BatchSpanProcessor(otlp_exporter)
)

# register trace provider
trace.set_tracer_provider(trace_provider)

Note that we’re setting a few configuration options using environment variables. You can set these as inline environment variables like this when starting your python app:

HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_DATASET=my-dataset \
SERVICE_NAME=my-favorite-service \
python app.py

Install the Honeycomb.OpenTelemetry nuget package using a NuGet package manager or the dotnet CLI.

For example, with the dotnet CLI, use:

dotnet add package Honeycomb.OpenTelemetry --prerelease

Configure your app for Honeycomb OpenTelemetry Distribution for .NET with one of the following configuration methods:

Programmatically  🔗

You can provide an HoneycombOptions object to programmatically configure the OpenTelemetry SDK.

using OpenTelemetry;

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(new HoneycombOptions
    {
        ServiceName = "my-app",
        ApiKey = "{apikey}",
        Dataset = "{dataset}"
    })
    .Build();

ASP.NET Core  🔗

You can provide an appsettings.json file:

{
  ...
  "Honeycomb": {
    "ServiceName": "my-app",
    "ApiKey": "{apikey}",
    "Dataset": "{dataset}",
  }
}

Then, pass the IConfiguration object to configure the OpenTelemetry SDK in your application startup code:

using Honeycomb.OpenTelemetry;

public IConfiguration Configuration { get; }

//...

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();

    // configure OpenTelemetry SDK to send data to Honeycomb
    services.AddHoneycomb(Configuration);
}

NOTE: It is possible to leverage the AppSettings ability to read environment variables to populate values in the configuration object. For example, the following environment variables could be used to override the values in an AppSettings file:

export HONEYCOMB__SERVICENAME=my-app
export HONEYCOMB__APIKEY={apikey}
export HONEYCOMB__DATASET={dataset}

See the OpenTelemetry ASP.NET Core instrumentation for more advanced options.

ASP.NET (.NET Framework v4.6.1+)  🔗

Add the following to your Web.config:

<system.webServer>
    <modules>
    <add name="TelemetryCorrelationHttpModule"
         type="Microsoft.AspNet.TelemetryCorrelation.TelemetryCorrelationHttpModule,Microsoft.AspNet.TelemetryCorrelation"
         preCondition="integratedMode,managedHandler" />
    </modules>
</system.webServer>

Configure the OpenTelemetry SDK to use Honeycomb during application start:

using OpenTelemetry;
using OpenTelemetry.Trace;
using Honeycomb.OpenTelemetry;

public class WebApiApplication : HttpApplication
{
    private TracerProvider _tracerProvider;

    protected void Application_Start()
    {
        _tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddHoneycomb(new HoneycombOptions
            {
                ServiceName = "my-app",
                ApiKey = "{apikey}",
                Dataset = "{dataset}"
            })
            .Build();
    }

    protected void Application_End()
    {
        _tracerProvider?.Dispose();
    }
}

See the OpenTelemetry ASP.NET instrumentation for more advanced options.

Command Line Arguments  🔗

You can provide command line arguments when starting your application:

dotnet run \
  --service-name=my-app \
  --honeycomb-apikey={apikey} \
  --honeycomb-dataset={dataset}

Then, pass the command line arguments to configure the OpenTelemetry SDK:

using OpenTelemetry;

//...

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(args)
    .Build();

NOTE: AddHoneycomb sets the OpenTelemetry SDK ResourceBuilder, which is used to create the Resource that is associated to all spans created by the tracer. Please do not override the ResourceBuilder by calling SetResourceBuilder with another instance. Otherwise, some resource attributes may not be queryable in your Honeycomb data.

The auto-instrumentation agent for Honeycomb OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is specified as a command-line argument when invoking the Java Virtual Machine (JVM).

In order to use the auto-instrumentation agent, you must first download and distribute the JAR file.

The agent JAR file must be available when running the java process that starts your service. The file can be downloaded from the GitHub Releases page or you can download it with a CLI tool like curl:

curl -LO https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/download/v0.7.0/honeycomb-opentelemetry-javaagent-0.7.0.jar

Once the JAR file is downloaded, you can configure the agent with Honeycomb credentials.

Java Agent Configuration  🔗

At a minimum, you need to configure the Honeycomb Agent with your API key and datasets. Choose from one of three configuration options.

  1. Configure using environment variables. For example:
SERVICE_NAME=my-favorite-service \
HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_TRACES_DATASET=my-traces \
HONEYCOMB_METRICS_DATASET=my-metrics \
java -javaagent:honeycomb-opentelemetry-javaagent-0.7.0.jar -jar /path/to/myapp.jar
  1. Configure using system properties. For example:
java \
-Dsample.rate=2 \
-Dservice.name=my-favorite-service \
-Dhoneycomb.api.key=my-api-key \
-Dhoneycomb.traces.dataset=my-traces \
-Dhoneycomb.metrics.dataset=my-metrics \
-javaagent:honeycomb-opentelemetry-javaagent-0.7.0.jar -jar /path/to/myapp.jar
  1. Configure using a properties file and either set an environment variable or system property. For example, in the myapp.properties file:
sample.rate=2
service.name=my-favorite-service
honeycomb.api.key=my-api-key
honeycomb.traces.dataset=my-traces
honeycomb.metrics.dataset=my-metrics

Use an environment variable:

HONEYCOMB_CONFIG_FILE=/path/to/file \
java -javaagent:honeycomb-opentelemetry-javaagent-0.7.0.jar -jar /path/to/myapp.jar

Or, use a system property:

java \
-Dhoneycomb.config.file=/path/to/file \
-javaagent:honeycomb-opentelemetry-javaagent-0.7.0.jar -jar /path/to/myapp.jar

The complete list of configuration options can be found at Advanced Configuration.

To auto-instrument your Node.js Express application, you need to add the following packages:

npm install --save @grpc/grpc-js
npm install --save @opentelemetry/api
npm install --save @opentelemetry/sdk-node
npm install --save @opentelemetry/exporter-trace-otlp-grpc
npm install --save @opentelemetry/auto-instrumentations-node

auto-instrumentations-node is a meta package that provides a simple way to initialize multiple Node.js instrumentations. Alternatively, you can install individual instrumentations.

Initialization  🔗

The OpenTelemetry initialization needs to happen first in your application lifecycle. You can accomplish this by creating an initialization file. You can then import the file as the first step in you application initialization, or include it with the -r, --require Node.js CLI option.

OpenTelemetry initialization:

// tracing.js

const process = require('process');
const { Metadata, credentials } = require("@grpc/grpc-js");

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");

const metadata = new Metadata()
metadata.set('x-honeycomb-team', '<YOUR_API_KEY>');
metadata.set('x-honeycomb-dataset', '<YOUR_DATASET>');
const traceExporter = new OTLPTraceExporter({
  url: 'grpc://api.honeycomb.io:443/',
  credentials: credentials.createSsl(),
  metadata
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: '<YOUR_SERVICE_NAME>',
  }),
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start()
  .then(() => console.log('Tracing initialized'))
  .catch((error) => console.log('Error initializing tracing', error));

process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Running the Node.js app with OpenTelemetry initialization file:

node -r ./tracing.js app.js

You can see a complete example here.

Follow the instructions below to set up the OpenTelemetry OTLP Exporter for Ruby.

Installation  🔗

To auto-instrument your Ruby application, add these gems to your Gemfile:

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-instrumentation-all'

The inclusion of opentelemetry-instrumentation-all in the above list provides instrumentations for Rails, Sinatra, several HTTP libraries, and more.

Ruby Initialization  🔗

The OpenTelemetry initialization needs to happen early in your application lifecycle. For Rails applications, the usual way to initialize OpenTelemetry is in a Rails initializer. For other Ruby services, perform this initialization as early as possible in the start-up process.

OpenTelemetry initialization:

# config/initializers/opentelemetry.rb
require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
require 'opentelemetry/instrumentation/all'
OpenTelemetry::SDK.configure do |c|
  c.use_all() # enables all instrumentation!
end

Environment configuration:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset"
export OTEL_SERVICE_NAME="your-service-name"

A complete example can be viewed here.

Choose the API and SDK for your language, or search the OpenTelemetry Registry for the term “instrumentation”.

Follow the instructions to install the required packages. Configure OpenTelemetry in your service as documented in the language’s OpenTelemetry SDK for an otlp GRPC exporter.

Or gather data from multiple services and control data egress, by exporting to an OpenTelemetry Collector on your network configured to forward onto Honeycomb.

Resources  🔗

Table of Contents