OpenTelemetry for Java | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

OpenTelemetry for Java

Honeycomb has an OpenTelemetry Distribution for Java that makes it easy to add automatic and manual instrumentation to your applications and have trace data sent to Honeycomb.

Automatic instrumentation is handled with a Java Agent that runs alongside your application. Additional manual instrumentation can be added using the OpenTelemetry API, which is available when using our SDK as a dependency.

The OpenTelemetry Java Agent supports many Java libraries and frameworks.

If you do not want to use automatic instrumentation, you will need to use the SDK Builder to instantiate an OpenTelemetryConfiguration instance. The Builder is not needed if the Agent is being used.

Requirements  🔗

These instructions will explain how to set up automatic and manual instrumentation for a service written in Java. In order to follow along, you will need:

  • Java version 8 or newer.
  • An application written in Java.
  • A Honeycomb API Key. You can find your API key on your Team Settings page. If you don’t have an API key yet, sign up for a free Honeycomb account.

Automatic Instrumentation with the OpenTelemetry Java Agent  🔗

The auto-instrumentation agent for Honeycomb OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is specified as a command-line argument when invoking the Java Virtual Machine (JVM).

In order to use the auto-instrumentation agent, you must first download and distribute the JAR file.

The agent JAR file must be available when running the java process that starts your service. The file can be downloaded from the GitHub Releases page or you can download it with a CLI tool like curl:

curl -LO https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/download/v0.8.1/honeycomb-opentelemetry-javaagent-0.8.1.jar

Once the JAR file is downloaded, you can configure the agent with Honeycomb credentials.

Configuration  🔗

At a minimum, you need to configure the Honeycomb Agent with your API key and datasets. Choose from one of three configuration options.

  1. Configure using environment variables. For example:
SERVICE_NAME=my-favorite-service \
HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_TRACES_DATASET=my-traces \
HONEYCOMB_METRICS_DATASET=my-metrics \
java -javaagent:honeycomb-opentelemetry-javaagent-0.8.1.jar -jar /path/to/myapp.jar
  1. Configure using system properties. For example:
java \
-Dsample.rate=2 \
-Dservice.name=my-favorite-service \
-Dhoneycomb.api.key=my-api-key \
-Dhoneycomb.traces.dataset=my-traces \
-Dhoneycomb.metrics.dataset=my-metrics \
-javaagent:honeycomb-opentelemetry-javaagent-0.8.1.jar -jar /path/to/myapp.jar
  1. Configure using a properties file and either set an environment variable or system property. For example, in the myapp.properties file:
sample.rate=2
service.name=my-favorite-service
honeycomb.api.key=my-api-key
honeycomb.traces.dataset=my-traces
honeycomb.metrics.dataset=my-metrics

Use an environment variable:

HONEYCOMB_CONFIG_FILE=/path/to/file \
java -javaagent:honeycomb-opentelemetry-javaagent-0.8.1.jar -jar /path/to/myapp.jar

Or, use a system property:

java \
-Dhoneycomb.config.file=/path/to/file \
-javaagent:honeycomb-opentelemetry-javaagent-0.8.1.jar -jar /path/to/myapp.jar

Advanced Configuration  🔗

This is the complete list of configuration options for the Honeycomb Agent. It can be configured using system properties or environment variables. System properties take precedence over environment variables if both are specified.

System Property /
Environment Variable
Default Value Description
honeycomb.api.key
HONEYCOMB_API_KEY
None [required – see note below] Your Honeycomb API key
honeycomb.traces.apikey
HONEYCOMB_TRACES_APIKEY
Value of honeycomb.api.key Your Honeycomb API key for sending traces
honeycomb.metrics.apikey
HONEYCOMB_METRICS_APIKEY
Value of honeycomb.api.key Your Honeycomb API key for sending metrics
honeycomb.dataset
HONEYCOMB_DATASET
None [required – see note below] Honeycomb dataset where traces will be sent
honeycomb.traces.dataset
HONEYCOMB_TRACES_DATASET
Value of honeycomb.dataset Honeycomb dataset where traces will be sent
honeycomb.metrics.dataset
HONEYCOMB_METRICS_DATASET
None Honeycomb dataset where traces will be sent (metrics will not be exported if this is not configured)
honeycomb.api.endpoint
HONEYCOMB_API_ENDPOINT
https://api.honeycomb.io:443 Honeycomb ingest endpoint.
honeycomb.traces.endpoint
HONEYCOMB_TRACES_ENDPOINT
Value of honeycomb.api.endpoint Honeycomb ingest endpoint for traces (defaults to the value of HONEYCOMB_API_ENDPOINT).
honeycomb.metrics.endpoint
HONEYCOMB_METRICS_ENDPOINT
Value of honeycomb.api.endpoint Honeycomb ingest endpoint for metrics (defaults to the value of HONEYCOMB_API_ENDPOINT).
sample.rate
SAMPLE_RATE
1 (retain all data) Sample rate for the deterministic sampler
service.name
SERVICE_NAME
None service.name attribute to be used for all spans

API key and dataset configuration options are required if you’re sending data to Honeycomb directly. If you’re using an OpenTelemetry Collector, you would specify your API key and dataset in the collector configuration file instead.

Adding Manual Instrumentation  🔗

Auto-instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add manual instrumentation where appropriate. You can use manual instrumentation whether you are using the Agent or the Builder.

To add manual instrumentation, add the Honeycomb OpenTelemetry SDK as a dependency in your project. This provides access to the OpenTelemetry APIs, which lets you access the currently executing span and add attributes to it, and/or to create new spans.

dependencies {
  implementation('io.honeycomb:honeycomb-opentelemetry-sdk:0.8.1')
}
<dependency>
  <groupId>io.honeycomb</groupId>
  <artifactId>honeycomb-opentelemetry-sdk</artifactId>
  <version>0.8.1</version>
</dependency>

Adding Attributes to Spans  🔗

It’s often beneficial to add attributes to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID.

In your code, import io.opentelemetry.api.trace.Span to get access to the span.

import io.opentelemetry.api.trace.Span;

...

Span span = Span.current();
span.setAttribute("user.id", user.getId());

This will add a user.id field to the current span so that you can use the field in WHERE, GROUP BY or ORDER clauses in the Honeycomb query builder.

Creating New Spans  🔗

Auto-instrumentation can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans. To do this, select the Tracer registered by the Agent and build a span.

In your code, import io.opentelemetry.api.OpenTelemetry, io.opentelemetry.api.trace.Span, and io.opentelemetry.api.trace.Tracer.

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

...

Tracer tracer = OpenTelemetry.getTracer("my-service");
Span span = tracer.spanBuilder("expensive-query").startSpan();

// ... do cool stuff

span.end();

Creating Spans Around Methods  🔗

You can also use the annotation @WithSpan to wrap the execution of a method with a span. The span will be automatically closed once the method has completed. Unless explicitly specified, the span will be named className.methodName. To override the name of the span, add a name in parentheses as an argument.

In your code, import io.opentelemetry.extension.annotations.WithSpan to allow usage of this annotation.

import io.opentelemetry.extension.annotations.WithSpan;

...

@WithSpan("importantSpan")
public String getImportantInfo() {
    return importantInfo;
}

Multi-span Attributes  🔗

Sometimes you’ll want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.

We’ll leverage the OpenTelemetry concept of baggage to do this. Baggage allows you to add a key with a value as an attribute to every subsequent child span of the current application context.

In your code, import io.opentelemetry.api.baggage.Baggage to allow use of the Baggage class.

import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.trace.Span;

...

Baggage.current()
  .toBuilder()
  .put("app.username", name)
  .build()
  .makeCurrent();

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

Sampling  🔗

Deterministic head sampling can be used with the Java Agent, with or without manual instrumentation.

The agent will read these variables and expect an integer that represents the sample rate you would like to apply. For example, a value of 5 means that one out of every five traces will be sent to Honeycomb.

To add sampling to the agent, specify the SAMPLE_RATE environment variable or the -Dsample.rate system property when invoking the Java Virtual Machine (JVM), as shown in Advanced Configuration.

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Metrics Collection  🔗

Honeycomb Enterprise accounts can also send metrics to Honeycomb.

JVM emits metrics at regular intervals that can be sent automatically to a Honeycomb dataset. These metrics will show up in data as runtime.jvm.gc.count, runtime.jvm.gc.time, runtime.jvm.memory.area, and so on.

Metrics are disabled by default. To enable metrics, set a HONEYCOMB_METRICS_DATASET environment variable. Optionally, set a specific HONEYCOMB_METRICS_APIKEY or HONEYCOMB_METRICS_ENDPOINT (if different from the traces API key and endpoint), as shown in Advanced Configuration.

Distributed Trace Propagation  🔗

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.

Using the Honeycomb SDK Builder  🔗

If you do not wish to use the auto-instrumentation agent, and/or wish to do more advanced instrumentation like adding other span processors or extensions, you can use the Honeycomb SDK to instantiate an OpenTelemetryConfiguration instance, which provides access to a Tracer and the rest of the OpenTelemetry API.

This should be configured as early as possible in the entry point of your application. Keep in mind, this builder is not required if the agent is in use.

Aside from requiring this builder, setting up manual instrumentation will be the same as when using the agent.

The OpenTelemetryConfiguration builder provides convenient methods for configuration:

import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.honeycomb.opentelemetry.sdk.trace.samplers.DeterministicTraceSampler; // optional
import io.honeycomb.opentelemetry.sdk.trace.spanprocessors.BaggageSpanProcessor; // optional

public OpenTelemetry honeycomb() {
  return OpenTelemetryConfiguration.builder()
    .setApiKey("your-api-key")
    .setDataset("your dataset")
    .setSampler(new DeterministicTraceSampler(2)) // optional - sends 1/2 of all traces generated
    .setServiceName("example-service")            // optional
    .addSpanProcessor(new BaggageSpanProcessor()) // optional
    .buildAndRegisterGlobal();
}

When using the Honeycomb SDK builder, specify a span processor and/or a sampler with a sample rate as an argument to the constructor.

The BaggageSpanProcessor is available as part of the Honeycomb SDK; it is what allows for multi-span attributes. The DeterministicTraceSampler is available as part of the Honeycomb SDK that samples based on a common field, given a sample rate.

The setSampler method accepts any instance of a class that implements the io.opentelemetry.sdk.trace.samplers.Sampler interface. Look at the provided deterministic sampler as an example of how to create your own custom sampler, if needed.

Using OpenTelemetry Without the Honeycomb Distribution  🔗

The primary purpose of Honeycomb’s Distribution for Java is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Java, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.

The Honeycomb Distribution reads specific variables and translates them to variables understood by upstream OpenTelemetry. For example, when using the distribution, the system property honeycomb.traces.endpoint is converted to the OpenTelemetry system property otel.exporter.otlp.traces.endpoint. Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.

When using OpenTelemetry for Java, all the following configuration properties are required:

System Property /
Environment Variable
Value
otel.traces.exporter
OTEL_TRACES_EXPORTER
otlp
otel.metrics.exporter
OTEL_METRICS_EXPORTER
otlp (*)
otel.exporter.otlp.endpoint
OTEL_EXPORTER_OTLP_ENDPOINT
https://api.honeycomb.io
otel.exporter.otlp.traces.endpoint
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
https://api.honeycomb.io (defaults to value of OTEL_EXPORTER_OTLP_ENDPOINT)
otel.exporter.otlp.metrics.endpoint
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
https://api.honeycomb.io (*)
otel.exporter.otlp.headers
OTEL_EXPORTER_OTLP_HEADERS
x-honeycomb-team=HONEYCOMB_API_KEY,x-honeycomb-dataset=HONEYCOMB_DATASET
otel.exporter.otlp.traces.headers
OTEL_EXPORTER_OTLP_TRACES_HEADERS
x-honeycomb-team=HONEYCOMB_API_KEY,x-honeycomb-dataset=HONEYCOMB_DATASET (defaults to value of OTEL_EXPORTER_OTLP_HEADERS)
otel.exporter.otlp.metrics.headers
OTEL_EXPORTER_OTLP_METRICS_HEADERS
x-honeycomb-team=HONEYCOMB_API_KEY,x-honeycomb-dataset=HONEYCOMB_DATASET (*)
otel.service.name
OTEL_SERVICE_NAME
service.name attribute to be used for all spans

Fields marked with an asterisk (*) are required for exporting metrics to Honeycomb.

More details on configuration options can be found on GitHub at OpenTelemetry SDK Autoconfigure.

If desired, a configuration file can be used to store environment variables or system properties for the Java Agent. A path to the configuration file must be provided using either a system property of otel.javaagent.configuration-file or an environment variable of OTEL_JAVAAGENT_CONFIGURATION_FILE.

For example, to specify a configuration file using system properties to run the agent with an app:

java -javaagent:path/to/opentelemetry-javaagent.jar \
     -Dotel.javaagent.configuration-file=path/to/config-file \
     -jar myapp.jar

Traces and Metrics Endpoint configuration options, as well as OpenTelemetry Headers, are listed in the OpenTelemetry for Java chart if sending data to Honeycomb directly.

If using an OpenTelemetry Collector, specify the endpoint of the collector, and add the headers to the collector configuration file.

Sampling Without the Honeycomb Distribution  🔗

System Property /
Environment Variable
Value
otel.traces.sampler
OTEL_TRACES_SAMPLER
traceidratio
otel.traces.sampler.arg
OTEL_TRACES_SAMPLER_ARG
0.5
otel.resource.attributes
OTEL_RESOURCE_ATTRIBUTES
SampleRate=2

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb re-weights sampled data, so it is recommended that you set a resource attribute containing the sample rate.

In the example above, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

Troubleshooting  🔗

Debug Mode  🔗

To enable debugging when running with the OpenTelemetry Java Agent, set the otel.javaagent.debug system property or OTEL_JAVAAGENT_DEBUG environment variable to true. When this setting is provided, the Agent configures a LoggingSpanExporter that logs traces & metrics data.

If you are not using the OpenTelemetry Java Agent, you can add a LoggingSpanExporter to your builder configuration. This will require adding another dependency on io.opentelemetry:opentelemetry-exporter-logging.

import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.opentelemetry.exporter.logging.LoggingSpanExporter; // for debugging
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; // for debugging

public OpenTelemetry honeycomb() {
  return OpenTelemetryConfiguration.builder()
    .setApiKey("your-api-key")
    .setDataset("your dataset")
    .addSpanProcessor(SimpleSpanProcessor.create(new LoggingSpanExporter())) // for debugging
    .buildAndRegisterGlobal();
}

Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.

gRPC transport customization  🔗

A gRPC transport is required to transmit OpenTelemetry data. HoneycombSDK includes grpc-netty-shaded.

If you’re using another gRPC dependency, version conflicts can come up with an error like this:

java.lang.NoSuchMethodError: io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51).

If you’d like to use another gRPC transport, you can exclude the grpc-netty-shaded transitive dependency:

dependencies {
    implementation('io.honeycomb:honeycomb-opentelemetry-sdk:0.5.0') {
        exclude group: 'io.grpc', module: 'grpc-netty-shaded'
    }
}
<project>
    <dependencies>
        <dependency>
            <groupId>io.honeycomb</groupId>
            <artifactId>honeycomb-opentelemetry-sdk</artifactId>
            <version>0.5.0</version>
            <exclusions>
                <exclusion>
                    <groupId>io.grpc</groupId>
                    <artifactId>grpc-netty-shaded</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</project>