OpenTelemetry for .NET | Honeycomb

Honeycomb.io Documentation
  1. Docs
  2. Send Data to Honeycomb
  3. OpenTelemetry for .NET

OpenTelemetry for .NET

Honeycomb has a OpenTelemetry Distribution for .NET that wraps the official OpenTelemetry .NET SDK to simplify the setup process and add instrumentation your applications to send telemetry data to Honeycomb.

Requirements 

These instructions explain how to instrument a service written in .NET. In order to follow along, you will need:

  • A .NET application that conforms to NET Standard 2.0, or a .NET Framework 4.6.2+ application.
  • A Honeycomb API Key.

You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Examples 

There are several examples that configure applications to send OpenTelemetry data to Honeycomb.

Automatic Instrumentation 

Acquire Dependencies 

Install the Honeycomb.OpenTelemetry NuGet package.

For example, with the .NET CLI, use:

dotnet add package Honeycomb.OpenTelemetry
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package Honeycomb.OpenTelemetry.CommonInstrumentations --prerelease

Initialize 

Configure your app for Honeycomb OpenTelemetry Distribution for .NET with one of the following configuration methods:

Initialize your ASP.NET Core app to use the Honeycomb OpenTelemetry Distribution for .NET:

  1. In your appsettings.json file, add a Honeycomb section:
{
  "Honeycomb": {
    "ServiceName": "my-app",
    "ApiKey": "{apikey}"
  }
}
  1. Get HoneycombOptions from configuration and use these options to configure the OpenTelemetry SDK in your application startup code:
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();

var honeycombOptions = builder.Configuration.GetHoneycombOptions();

// Set up OpenTelemetry Tracing
builder.Services.AddOpenTelemetry().WithTracing(otelBuilder =>
{
    otelBuilder
        .AddHoneycomb(honeycombOptions)
        .AddCommonInstrumentations();
});

// Register a Tracer, so it can be injected into other components (for example, Controllers)
builder.Services.AddSingleton(TracerProvider.Default.GetTracer(honeycombOptions.ServiceName));

var app = builder.Build();

app.MapGet("/", (Tracer tracer) =>
{
    using var span = tracer.StartActiveSpan("app.manual-span");
    span.SetAttribute("app.manual-span.message", "Adding custom spans is also super easy!");
});

await app.RunAsync();
Note

If you are a Honeycomb Classic user, you must also specify the Dataset using the Dataset field in the appsettings.json file. A Dataset is a bucket where data gets stored in Honeycomb.

{
  "Honeycomb": {
    "ServiceName": "my-app",
    "ApiKey": "{apikey}",
    "Dataset": "{dataset}"
  }
}

To programmatically configure the OpenTelemetry SDK, provide a HoneycombOptions object:

using OpenTelemetry;
using Honeycomb.OpenTelemetry;

var options = new HoneycombOptions
{
    ServiceName = "my-app",
    ServiceVersion = "1.0.0",
    ApiKey = "{apikey}",
    ResourceBuilder = ResourceBuilder.CreateDefault().AddAttributes(
        new Dictionary<string, object>
        {
            {"custom-resource-attribute", "some-value"}
        })
};

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(options)
    .Build();
Note

If you are a Honeycomb Classic user, you must also specify the Dataset using the Dataset field in the HoneycombOptions object. A Dataset is a bucket where data gets stored in Honeycomb.

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(new HoneycombOptions
    {
        ServiceName = "my-app",
        ApiKey = "{apikey}"
        Dataset = "{dataset}"
    })
    .Build();

Add the following to your Web.config:

<system.webServer>
    <modules>
        <add
            name="TelemetryHttpModule"
            type="OpenTelemetry.Instrumentation.AspNet.TelemetryHttpModule,
                OpenTelemetry.Instrumentation.AspNet.TelemetryHttpModule"
            preCondition="integratedMode,managedHandler" />
    </modules>
</system.webServer>

Configure the OpenTelemetry SDK to use Honeycomb during application start:

using OpenTelemetry;
using OpenTelemetry.Trace;
using Honeycomb.OpenTelemetry;

public class WebApiApplication : HttpApplication
{
    private TracerProvider _tracerProvider;

    protected void Application_Start()
    {
        _tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddHoneycomb(new HoneycombOptions
            {
                ServiceName = "my-app",
                ApiKey = "{apikey}"
            })
            .Build();
    }

    protected void Application_End()
    {
        _tracerProvider?.Dispose();
    }
}

For more advanced configuration options, see the OpenTelemetry ASP.NET instrumentation.

Note

If you are a Honeycomb Classic user, you must also specify the Dataset using the Dataset field in the HoneycombOptions object. A Dataset is a bucket where data gets stored in Honeycomb.

_tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(new HoneycombOptions
    {
        ServiceName = "my-app",
        ApiKey = "{apikey}",
        Dataset = "{dataset}"
    })
    .Build();

Using HTTP/protobuf instead of gRPC 

By default, the OpenTelemetry .NET SDK uses the gRPC protocol to send telemetry data. To use HTTP instead of gRPC for traces, update the protocol in the OtlpExporter:

services.AddOpenTelemetry().WithTracing(builder => builder
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(option =>
    {
        option.Endpoint = new Uri("https://api.honeycomb.io/v1/traces");
        option.Headers = $"x-honeycomb-team={honeycombApiKey}";
        option.Protocol = OtlpExportProtocol.HttpProtobuf;
    }));

Note that the endpoint URL is different for HTTP/protobuf than for gRPC in that it specifies the signal type (/v1/traces).

If the application is running on .NET Core 3.x, also add the following at application startup:

AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
Note

Endpoint configuration options, as well as OpenTelemetry Headers, are listed in the OpenTelemetry for .NET chart if sending data to Honeycomb directly.

If using an OpenTelemetry Collector, specify the endpoint of the collector, and add the headers to the collector configuration file.

Using Instrumentation Packages 

By default, the Honeycomb Distribution does not bundle any instrumentation packages. However, two additional packages exist to make getting started easier:

  • Honeycomb.OpenTelemetry.Instrumentation.AspNetCore
  • Honeycomb.OpenTelemetry.CommonInstrumentations

The Honeycomb.OpenTelemetry.Instrumentation.AspNetCore package wraps the existing ASP.NET Core instrumentation package published by OpenTelemetry, but also copies the current trace baggage into spans created on incoming requests. If you’d prefer not to have this behavior, you can use the OpenTelemetry AspNetCore instrumentation package directly.

To use Honeycomb.OpenTelemetry.Instrumentation.AspNetCore, first add the package:

dotnet add package Honeycomb.OpenTelemetry.Instrumentation.AspNetCore --prerelease

Then, configure the OpenTelemetry SDK to add ASP.NET Core instrumentation during application start:

using OpenTelemetry.Trace;

var honeycombOptions = builder.Configuration.GetHoneycombOptions();

builder.Services.AddOpenTelemetry().WithTracing(otelBuilder =>
    otelBuilder
        .AddHoneycomb(honeycombOptions)
        .AddAspNetCoreInstrumentationWithBaggage()
);

The Honeycomb.OpenTelemetry.CommonInstrumentations package bundles several common instrumentation libraries:

For .NET 6 and above:

  • Honeycomb.OpenTelemetry.Instrumentation.AspNetCore
  • OpenTelemetry.Instrumentation.GrpcNetClient

For .NET Framework 4.6.2 and above:

  • OpenTelemetry.Instrumentation.Owin
  • OpenTelemetry.Instrumentation.AspNet

For .NET or .NET Framework 4.7.1 and above:

  • OpenTelemetry.Instrumentation.Quartz

And for all .NET flavors and versions:

  • Npgsql.OpenTelemetry
  • OpenTelemetry.Instrumentation.EntityFrameworkCore
  • OpenTelemetry.Instrumentation.Http
  • OpenTelemetry.Instrumentation.MySqlData
  • OpenTelemetry.Instrumentation.SqlClient
  • OpenTelemetry.Instrumentation.StackExchangeRedis
  • OpenTelemetry.Instrumentation.Wcf

To use the CommonInstrumentations package, first add the package:

dotnet add package Honeycomb.OpenTelemetry.CommonInstrumentations --prerelease

Then, configure the OpenTelemetry SDK to add instrumentation during application start:

using OpenTelemetry.Trace;

var honeycombOptions = builder.Configuration.GetHoneycombOptions();

builder.Services.AddOpenTelemetry().WithTracing(otelBuilder =>
    otelBuilder
        .AddHoneycomb(honeycombOptions)
        .AddCommonInstrumentations();
);

Additional instrumentation packages can be added during initialization. For example, to add the AWS Lambda Instrumentation package from OpenTelemetry-Contrib, install the package:

dotnet add package OpenTelemetry.Instrumentation.AWSLambda --prerelease

Then, configure the OpenTelemetry SDK to add AWS instrumentation during application start:

using OpenTelemetry;

//...

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb()
    .AddAWSLambdaConfigurations()
    .Build();
Note
Redis instrumentation requires access to the IConnection object to setup tracing. This can either be provided as part of the AddHoneycomb call by setting the RedisConnection property on HoneycombOptions or it can be automatically retrieved if it is registered in the ASP.NET ServiceCollection object.

Adding Manual Instrumentation 

Acquiring a Tracer 

To create spans, you need to acquire a Tracer.

using OpenTelemetry.Trace;

//...

var tracer = TracerProvider.Default.GetTracer("tracer.name.here");

Then, inject the Tracer instance with ASP.NET Core dependency injection or manage its lifecycle manually.

When you create a Tracer, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter.

When traces are sent to Honeycomb, the name of the Tracer is turned into the library.name field, which can be used to show all spans created from a particular tracer.

In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.

The library.name field is also used with traces created from instrumentation libraries.

Adding Attributes to Spans 

It is often beneficial to add attributes to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID:

using OpenTelemetry.Trace;

//...

var currentSpan = Tracer.CurrentSpan;
currentSpan.SetAttribute("user.id", User.GetUserId())

This configuration will add a user.id attribute to the current span, so that you can use the field in WHERE, GROUP BY, or ORDER clauses in the Honeycomb query builder.

Creating New Spans 

In order to get the full picture of what is happening, you will have to add manual instrumentation and create custom spans to describe what is happening in your application. To do this, grab your tracer instance and use it to create a span:

using OpenTelemetry.Trace;

//...

using var span = TracerProvider.Default.GetTracer("my-service").StartActiveSpan("expensive-query")
// ... do cool stuff

Multi-Span Attributes 

Sometimes you want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.

We will leverage the OpenTelemetry concept of baggage to add this attribute. Baggage allows you to add a key with a value as an attribute to every subsequent child span of the current application context.

using OpenTelemetry;

...

Baggage.Current.SetBaggage("app.username", name);

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

More on Manual Instrumentation 

The OpenTelemetry documentation for .NET has a comprehensive set of topics on manual instrumentation, both with the System.Diagnostics API and the OpenTelemetry Shim.

Sending Data to a Collector or Other Endpoint 

You can send data to an OpenTelemetry Collector instance or another endpoint by specifying a different endpoint.

For example, here is how to send to an OpenTelemetry Collector instance over HTTP:

var options = new HoneycombOptions
{
    ServiceName = "my-app",
    ApiKey = "{apikey}"
    TracesEndpoint = "http://<your-collector-endpoint>/v1/traces"

    // MetricsEndpoint is not required if you are not sending metrics
    MetricsEndpoint = "http://<your-collector-endpoint>/v1/metrics"
}

{
  "Honeycomb": {
    "TracesEndpoint": "http://<your-collector-endpoint>/v1/traces",
    "MetricsEndpoint": "http://<your-collector-endpoint>/v1/metrics"
  }
}

Sampling 

Deterministic sampling can be enabled by providing a SampleRate via the following methods. A sample rate of 5 means that one out of every five traces will be sent to Honeycomb.

var options = new HoneycombOptions
{
    ServiceName = "my-app",
    ApiKey = "{apikey}"
    SampleRate = 5 // sends 1/5 traces
}

The value of SampleRate must be a positive integer.

{
  "Honeycomb": {
    "SampleRate": 5
  }
}

The value of SampleRate must be a positive integer.

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Visualize Traces Locally 

Honeycomb’s OpenTelemetry Distribution for .NET can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation.

Enable local visualizations through application settings or environment variables.

In appsettings.json, set EnableLocalVisualizations to true.

{
  "Honeycomb": {
    "EnableLocalVisualizations": true
  }
}

Set the HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS environment variable to true.

export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true

Then, run your application.

dotnet run

The output displays the name of the root span and a link to Honeycomb that shows its trace. For example:

Trace for root-span-name
Honeycomb link: <link to Honeycomb trace>
Note
In production, disable local visualization as it creates additional overhead to create the link and print it to the console.

Using OpenTelemetry Without the Honeycomb Distribution 

The primary purpose of Honeycomb’s SDK Distribution for .NET is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb SDK Distribution is using OpenTelemetry for .NET, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the the Honeycomb SDK Distribution for .NET.

The Honeycomb SDK Distribution reads specific variables and translates them to variables understood by upstream OpenTelemetry. For example, when using the distribution, the environment variable HONEYCOMB_API_ENDPOINT is converted to the OpenTelemetry variable OTEL_EXPORTER_OTLP_ENDPOINT. Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.

When using OpenTelemetry for .NET, the following configuration properties are required:

OtlpExporterOptions property /
Environment Variable		 Value
Endpoint
OTEL_EXPORTER_OTLP_ENDPOINT		 https://api.honeycomb.io
Headers
OTEL_EXPORTER_OTLP_HEADERS		 x-honeycomb-team=HONEYCOMB_API_KEY
ServiceName
OTEL_SERVICE_NAME		 your-service-name

The service name must be set in either the TracerProvider configuration code or as an environment variable.

The code below configures a tracer provider that can export traces to Honeycomb.

services.AddOpenTelemetry().WithTracing(builder => builder
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(option =>
    {
        option.Endpoint = new Uri("https://api.honeycomb.io");
        option.Headers = $"x-honeycomb-team={honeycombApiKey}";
    }));

More details on configuration options can be found on GitHub at OpenTelemetry for .NET.

Sampling Without the Honeycomb Distribution 

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.

Use a TraceIdRatioBased sampler, with a ratio expressed as 1/N. Then, also create a resource attribute called SampleRate with the value of N. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.

In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

services.AddOpenTelemetry().WithTracing((builder) => builder
    .SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService(serviceName)
        // IMPORTANT: add a SampleRate of 2 as a resource attribute
        .AddAttributes(new[] { new KeyValuePair<string, object>("SampleRate", 2) })
    )
    .SetSampler(new TraceIdRatioBasedSampler(0.5)) // sampler
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(option =>
    {
        option.Endpoint = new Uri("https://api.honeycomb.io");
        option.Headers = $"x-honeycomb-team={honeycombApiKey}";
    }));

Using HTTP Instead of gRPC 

By default, the OpenTelemetry .NET SDK uses the gRPC protocol to send telemetry data. To use HTTP instead of gRPC for traces, update the protocol in the OtlpExporter:

services.AddOpenTelemetry().WithTracing(builder => builder
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(option =>
    {
        option.Endpoint = new Uri("https://api.honeycomb.io/v1/traces");
        option.Headers = $"x-honeycomb-team={honeycombApiKey}";
        option.Protocol = OtlpExportProtocol.HttpProtobuf;
    }));

Note that the endpoint URL is different for HTTP/protobuf than for gRPC in that it specifies the signal type (/v1/traces).

If the application is running on .NET Core 3.x, also add the following at application startup:

AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
Note

If you are sending data directly to Honeycomb, you can find Endpoint configuration options and OpenTelemetry Headers in the OpenTelemetry for .NET chart.

If you are using an OpenTelemetry Collector, specify the endpoint of the collector, and add the headers to the collector configuration file.

Endpoint URLs for OTLP/HTTP 

When using the OTEL_EXPORTER_OTLP_ENDPOINT environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.

For example, when exporting trace data, the endpoint is updated to append v1/traces. When exporting metrics data, the endpoint is updated to append v1/metrics. The same modification is not necessary for gRPC.

By setting OTEL_EXPORTER_OTLP_ENDPOINT to https://api.honeycomb.io, traces are sent to https://api.honeycomb.io/v1/traces and metrics to https://api.honeycomb.io/v1/metrics.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io

If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT.

When using a signal-specific environment variable, these paths must be appended manually. Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces, appending the endpoint with v1/traces, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT for metrics, appending the endpoint with v1/metrics.

Send both traces and metrics to Honeycomb using this method by setting the following variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics

More details about endpoints and signals can be found in the OpenTelemetry Specification.

Troubleshooting 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

My Traces have Duplicate Spans 

If a This trace has multiple spans sharing the same non-null span ID error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.

One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.

Error Log 

By default, the .NET SDK does not emit error information to the console. The SDK does include a self-diagnostics feature to help with troubleshooting by writing errors to a log file. Enable self-diagnostics by creating a file in the current working directory called OTEL_DIAGNOSTICS.json:

{
    "LogDirectory": ".",
    "FileSize": 1024,
    "LogLevel": "Error"
}

Running the app with the above configuration in the OTEL_DIAGNOSTICS.json file will generate a file named for ExecutableName.ProcessId.log, such as console.30763.log. The newly generated log file will contain any applicable errors for the app. The LogDirectory represents the directory in which the log file will be stored, and can be changed to output to a different location. The FileSize is the maximum size the log file can grow in KiB, and can be adjusted if a larger size is needed to prevent overwriting of logging output. Adjust the log level as needed for more or less verbose logging, using the fields available with System Diagnostics.

To disable this error log, delete the OTEL_DIAGNOSTICS.json file.

Exporting to the Console 

If you are using the Honeycomb OpenTelemetry Distribution, warnings will appear in the console if you are missing an API Key or Dataset. If no errors appear but your data is not in Honeycomb as expected, use a ConsoleExporter to print your spans to the console. This will help confirm whether your app is being instrumented with the data you expect.

First, import the ConsoleExporter:

dotnet add package OpenTelemetry.Exporter.Console --prerelease

Then add the ConsoleExporter to your configuration:

using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
    .AddHoneycomb(options)
    .AddConsoleExporter() // for debugging
    .Build();

Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.

Receiving 464 Errors 

You may receive a 464 error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.

Instrumentation Package Rename 

Previous versions of the Honeycomb OpenTelemetry Distribution had a package called Honeycomb.OpenTelemetry.AutoInstrumentations, which was renamed to Honeycomb.OpenTelemetry.CommonInstrumentations. The name change of the instrumentation package is intended to better reflect the purpose of the package, but is a potential upgrade step to ensure the appropriate package is being used. Refer to our Releases page on GitHub as needed.

OpenTelemetry Tracing and Metrics Rename 

Previous versions of OpenTelemetry used AddOpenTelemetryTracing and AddOpenTelemetryMetrics to add tracing and metrics. AddOpenTelemetryTracing has been replaced with AddOpenTelemetry().WithTracing. AddOpenTelemetryMetrics has been replaced with AddOpenTelemetry().WithMetrics.

On this page: