1. Docs
2. Send Data to Honeycomb
3. OpenTelemetry for Java

OpenTelemetry for Java

Table of Contents

Honeycomb has an OpenTelemetry Distribution for Java that makes it easy to add automatic and manual instrumentation to your applications and have trace data sent to Honeycomb.

Automatic instrumentation is handled with a Java Agent that runs alongside your application. Additional manual instrumentation can be added using the OpenTelemetry API, which is available when using our SDK as a dependency.

The OpenTelemetry Java Agent supports many Java libraries and frameworks.

If you do not want to use automatic instrumentation, you will need to use the SDK Builder to instantiate an OpenTelemetryConfiguration instance. The Builder is not needed if the Agent is being used.

Requirements 

These instructions will explain how to set up automatic and manual instrumentation for a service written in Java. In order to follow along, you will need:

• Java version 8 or newer.
• An application written in Java.
• A Honeycomb API Key. You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Automatic Instrumentation with the OpenTelemetry Java Agent 

Acquire Dependencies 

The automatic instrumentation agent for Honeycomb OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is run alongside your app.

In order to use the automatic instrumentation agent, you must first download it:

curl -LO https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/download/v1.5.2/honeycomb-opentelemetry-javaagent-1.5.2.jar

Configure 

Choose from one of the following configuration options:

Use environment variables:

SERVICE_NAME=my-favorite-service
HONEYCOMB_API_KEY=my-api-key
HONEYCOMB_METRICS_DATASET=my-metrics

Or, use system properties:

sample.rate=2
service.name=my-favorite-service
honeycomb.api.key=my-api-key
honeycomb.metrics.dataset=my-metrics

and set an environment variable with a path to your properties file:

HONEYCOMB_CONFIG_FILE=/path/to/properties/file

# environment variable
HONEYCOMB_TRACES_DATASET=my-traces

# system property
honeycomb.traces.dataset=my-traces

Run 

Run your application with the automatic instrumentation agent as a sidecar:

java -javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar -jar /path/to/myapp.jar

You can also include configuration values with an invocation of your application:

java \
-Dhoneycomb.config.file=/path/to/properties/file \
-javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar -jar /path/to/myapp.jar

For more configuration options, read Honeycomb OpenTelemetry Distribution for Java.

In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.

Advanced Configuration 

This is the complete list of configuration options for the Honeycomb Agent. It can be configured using system properties or environment variables. System properties take precedence over environment variables if both are specified.

For all other agent configuration, refer to the OpenTelemetry documentation on the Java Agent Configuration. This contains details about configuring and suppressing specific instrumentations.

Using HTTP/protobuf instead of gRPC 

By default, the Honeycomb OpenTelemetry SDK uses the gRPC protocol to send telemetry data. To use HTTP instead of gRPC, update the protocol using one of the configuration methods:

• System property: -Dotel.exporter.otlp.protocol=http/protobuf
• Environment variable: export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Adding Manual Instrumentation 

Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add manual instrumentation where appropriate. You can use manual instrumentation whether you are using the Agent or the Builder.

To add manual instrumentation, add the Honeycomb OpenTelemetry SDK as a dependency in your project. This provides access to the OpenTelemetry APIs, which lets you access the currently executing span and add attributes to it, and/or to create new spans.

dependencies {
  implementation('io.honeycomb:honeycomb-opentelemetry-sdk:1.5.2')
}

<dependency>
  <groupId>io.honeycomb</groupId>
  <artifactId>honeycomb-opentelemetry-sdk</artifactId>
  <version>1.5.2</version>
</dependency>

Acquiring a Tracer 

To create spans, you need to get a Tracer.

import io.opentelemetry.api;

//...

Tracer tracer = openTelemetry.getTracer("tracer.name.here");

When you create a Tracer, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter.

When traces are sent to Honeycomb, the name of the Tracer is turned into the library.name field, which can be used to show all spans created from a particular tracer.

In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.

The library.name field is also used with traces created from instrumentation libraries.

Adding Attributes to Spans 

It is often beneficial to add attributes to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your service in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID.

In your code, import io.opentelemetry.api.trace.Span to get access to the span.

import io.opentelemetry.api.trace.Span;

...

Span span = Span.current();
span.setAttribute("user.id", user.getId());

This will add a user.id field to the current span so that you can use the field in WHERE, GROUP BY or ORDER clauses in the Honeycomb query builder.

Creating New Spans 

Automatic instrumentation can show the shape of requests to your system, but only you know the really important parts. In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans. To do this, select the Tracer registered by the Agent and build a span.

In your code, import io.opentelemetry.api.GlobalOpenTelemetry, io.opentelemetry.api.trace.Span, and io.opentelemetry.api.trace.Tracer.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

...

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");
Span span = tracer.spanBuilder("expensive-query").startSpan();

// ... do cool stuff

span.end();

Creating Spans Around Methods 

You can also use the annotation @WithSpan to wrap the execution of a method with a span. The span will be automatically closed once the method has completed. Unless explicitly specified, the span will be named className.methodName. To override the name of the span, add a name in parentheses as an argument.

In your code, import io.opentelemetry.instrumentation.annotations.WithSpan to allow usage of this annotation.

import io.opentelemetry.instrumentation.annotations.WithSpan;

...

@WithSpan("importantSpan")
public String getImportantInfo() {
    return importantInfo;
}

Multi-Span Attributes 

Sometimes you will want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.

We will leverage the OpenTelemetry concept of baggage to do this. Baggage allows you to add a key with a value as an attribute to every subsequent child span of the current application context.

In your code, import io.opentelemetry.api.baggage.Baggage to allow use of the Baggage class.

import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.context.Scope;

...

try (final Scope ignored = Baggage.current()
    .toBuilder()
    .put("app.username", name)
    .build()
    .makeCurrent()
) {
    // all subsequently created span in this block will have the `app.username` attribute
}

Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

More on Manual Instrumentation 

The OpenTelemetry documentation for Java has a comprehensive set of topics on manual instrumentation and the Annotations API.

Sampling 

Deterministic head sampling can be used with the Java Agent, with or without manual instrumentation.

The agent will read these variables and expect an integer that represents the sample rate you would like to apply. For example, a value of 5 means that one out of every five traces will be sent to Honeycomb.

To add sampling to the agent, specify the SAMPLE_RATE environment variable or the -Dsample.rate system property when invoking the Java Virtual Machine (JVM), as shown in Advanced Configuration.

The value of your sample rate must be a positive integer.

If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.

Metrics Collection 

The JVM emits metrics at regular intervals that can be sent automatically to a Honeycomb dataset. These metrics will show up in data as runtime.jvm.gc.count, runtime.jvm.gc.time, runtime.jvm.memory.area, and so on. If an instrumentation library also supports emitting metrics, then those metrics will be sent as well.

Metrics are currently disabled by default. To enable metrics, set a HONEYCOMB_METRICS_DATASET environment variable. Optionally, set a specific HONEYCOMB_METRICS_APIKEY or HONEYCOMB_METRICS_ENDPOINT (if different from the traces API key and endpoint), as shown in Advanced Configuration.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Using the Honeycomb SDK Builder 

If you do not wish to use the automatic instrumentation agent, and/or wish to do more advanced instrumentation like adding other span processors or extensions, you can use the Honeycomb SDK to instantiate an OpenTelemetryConfiguration instance, which provides access to a Tracer and the rest of the OpenTelemetry API.

This should be configured as early as possible in the entry point of your application. Keep in mind, this builder is not required if the agent is in use.

Aside from requiring this builder, setting up manual instrumentation will be the same as when using the agent.

The OpenTelemetryConfiguration builder provides convenient methods for configuration:

import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.honeycomb.opentelemetry.sdk.trace.samplers.DeterministicTraceSampler; // optional
import io.honeycomb.opentelemetry.sdk.trace.spanprocessors.BaggageSpanProcessor; // optional

public OpenTelemetry honeycomb() {
  return OpenTelemetryConfiguration.builder()
    .setApiKey("your-api-key")
    .setServiceName("your-service-name")
    .setOtlpProtocol("http/protobuf") // optional - use HTTP instead of gRPC to transmit telemetry
    .setSampler(new DeterministicTraceSampler(2)) // optional - sends 1/2 of all traces generated
    .addSpanProcessor(new BaggageSpanProcessor()) // optional
    .buildAndRegisterGlobal();
}

When using the Honeycomb SDK builder, specify a span processor and/or a sampler with a sample rate as an argument to the constructor.

The BaggageSpanProcessor is available as part of the Honeycomb SDK; it is what allows for multi-span attributes. The DeterministicTraceSampler is available as part of the Honeycomb SDK that samples based on a common field, given a sample rate.

The setSampler method accepts any instance of a class that implements the io.opentelemetry.sdk.trace.samplers.Sampler interface. Look at the provided deterministic sampler as an example of how to create your own custom sampler, if needed.

Include the Java Agent as Part of Build Process 

Another option for using the agent is to include it when running Maven targets or Gradle tasks. For example, here is a possible configuration to attach the Java agent when running a Spring Boot app:

bootRun {
    environment = ["HONEYCOMB_API_KEY": System.getenv("HONEYCOMB_API_KEY"),
                   "OTEL_SERVICE_NAME": System.getenv("OTEL_SERVICE_NAME")]
    jvmArgs = ["-javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar"]
}

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <environmentVariables>
                        <OTEL_SERVICE_NAME>${env.OTEL_SERVICE_NAME}</OTEL_SERVICE_NAME>
                        <HONEYCOMB_API_KEY>${env.HONEYCOMB_API_KEY}</HONEYCOMB_API_KEY>
                    </environmentVariables>
                    <agents>
                        <agent>
                            honeycomb-opentelemetry-javaagent-1.5.2.jar
                        </agent>
                    </agents>
                </configuration>
            </plugin>
        </plugins>
    </build>

Be sure to set the environment variables for HONEYCOMB_API_KEY and OTEL_SERVICE_NAME as described in the Configure step above.

Using OpenTelemetry Without the Honeycomb Distribution 

The primary purpose of Honeycomb’s Distribution for Java is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Java, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.

The Honeycomb Distribution reads specific variables and translates them to variables understood by upstream OpenTelemetry. For example, when using the distribution, the system property honeycomb.traces.endpoint is converted to the OpenTelemetry system property otel.exporter.otlp.traces.endpoint. Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.

As per the OpenTelemetry specification, it is also required to set a service.name resource in your SDK configuration. The service name is used as the name of a dataset to store trace data in Honeycomb.

When using OpenTelemetry for Java, all the following configuration properties are required:

Fields marked with an asterisk (*) are required for exporting metrics to Honeycomb.

If you do not have a service name set in TracerProvider configuration code, you must also set the OTEL_SERVICE_NAME environment variable.

More details on configuration options can be found on GitHub at OpenTelemetry SDK Autoconfigure.

If desired, a configuration file can be used to store environment variables or system properties for the Java Agent. A path to the configuration file must be provided using either a system property of otel.javaagent.configuration-file or an environment variable of OTEL_JAVAAGENT_CONFIGURATION_FILE.

For example, to specify a configuration file using system properties to run the agent with an app:

java -javaagent:path/to/opentelemetry-javaagent.jar \
     -Dotel.javaagent.configuration-file=path/to/config-file \
     -jar myapp.jar

Using HTTP Instead of gRPC 

By default, OpenTelemetry for Java uses gRPC protocol. To use HTTP instead of gRPC, update the protocol using one of the configuration methods:

• System property: -Dotel.exporter.otlp.protocol=http/protobuf
• Environment variable: export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

The protocol can also be set specific to each signal, such as OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL.

Sampling Without the Honeycomb Distribution 

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.

Use a TraceIdRatioBased sampler, with a ratio expressed as 1/N. Then, also create a resource attribute called SampleRate with the value of N. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.

In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

The value of SampleRate must be a positive integer.

Endpoint URLs for OTLP/HTTP 

When using the OTEL_EXPORTER_OTLP_ENDPOINT environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.

For example, when exporting trace data, the endpoint is updated to append v1/traces. When exporting metrics data, the endpoint is updated to append v1/metrics. The same modification is not necessary for gRPC.

By setting OTEL_EXPORTER_OTLP_ENDPOINT to https://api.honeycomb.io, traces are sent to https://api.honeycomb.io/v1/traces and metrics to https://api.honeycomb.io/v1/metrics.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io

If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT.

When using a signal-specific environment variable, these paths must be appended manually. Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for traces, appending the endpoint with v1/traces, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT for metrics, appending the endpoint with v1/metrics.

Send both traces and metrics to Honeycomb using this method by setting the following variables:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics

More details about endpoints and signals can be found in the OpenTelemetry Specification.

Troubleshooting 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

My Traces have Duplicate Spans 

If a This trace has multiple spans sharing the same non-null span ID error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.

One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.

Debug Mode 

To enable debugging when running with the OpenTelemetry Java Agent, set the otel.javaagent.debug system property or OTEL_JAVAAGENT_DEBUG environment variable to true. When this setting is provided, the Agent configures a LoggingSpanExporter that logs traces & metrics data.

If you are not using the OpenTelemetry Java Agent, you can add a LoggingSpanExporter to your builder configuration. This will require adding another dependency on io.opentelemetry:opentelemetry-exporter-logging.

import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.opentelemetry.exporter.logging.LoggingSpanExporter; // for debugging
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; // for debugging

public OpenTelemetry honeycomb() {
  return OpenTelemetryConfiguration.builder()
    .setApiKey("your-api-key")
    .setServiceName("your-service-name")
    .addSpanProcessor(SimpleSpanProcessor.create(new LoggingSpanExporter())) // for debugging
    .buildAndRegisterGlobal();
}

Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.

gRPC Transport Customization 

A gRPC transport is required to transmit OpenTelemetry data. HoneycombSDK includes grpc-netty-shaded.

If you are using another gRPC dependency, version conflicts can come up with an error like this:

java.lang.NoSuchMethodError: io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51).

If you would like to use another gRPC transport, you can exclude the grpc-netty-shaded transitive dependency:

dependencies {
  implementation('io.honeycomb:honeycomb-opentelemetry-sdk:1.5.2') {
    exclude group: 'io.grpc', module: 'grpc-netty-shaded'
  }
}

<dependency>
  <groupId>io.honeycomb</groupId>
  <artifactId>honeycomb-opentelemetry-sdk</artifactId>
  <version>1.5.2</version>

  <exclusions>
    <exclusion>
      <groupId>io.grpc</groupId>
      <artifactId>grpc-netty-shaded</artifactId>
    </exclusion>
  </exclusions>

</dependency>

Receiving 464 Errors 

You may receive a 464 error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.

Additionally, older JVMs may not have sufficient gRPC support and may attempt to send telemetry using HTTP1. To resolve this, either update to a newer JVM or use http/protobuf as the transfer protocol.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit