Honeycomb has an OpenTelemetry Distribution for Java that makes it easy to add automatic and manual instrumentation to your applications and have trace data sent to Honeycomb.
Automatic instrumentation is handled with a Java Agent that runs alongside your application. Additional manual instrumentation can be added using the OpenTelemetry API, which is available when using our SDK as a dependency.
The OpenTelemetry Java Agent supports many Java libraries and frameworks.
If you do not want to use automatic instrumentation, you will need to use the SDK Builder to instantiate an OpenTelemetryConfiguration
instance.
The Builder is not needed if the Agent is being used.
These instructions will explain how to set up automatic and manual instrumentation for a service written in Java. In order to follow along, you will need:
The automatic instrumentation agent for Honeycomb OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is run alongside your app.
In order to use the automatic instrumentation agent, you must first download it:
curl -LO https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/download/v1.5.2/honeycomb-opentelemetry-javaagent-1.5.2.jar
Choose from one of the following configuration options:
Use environment variables:
SERVICE_NAME=my-favorite-service
HONEYCOMB_API_KEY=my-api-key
HONEYCOMB_METRICS_DATASET=my-metrics
Or, use system properties:
sample.rate=2
service.name=my-favorite-service
honeycomb.api.key=my-api-key
honeycomb.metrics.dataset=my-metrics
and set an environment variable with a path to your properties file:
HONEYCOMB_CONFIG_FILE=/path/to/properties/file
If you are a Honeycomb Classic user, the Dataset also must be specified using the HONEYCOMB_TRACES_DATASET
environment variable or honeycomb.traces.dataset
system property.
A Dataset is a bucket where data gets stored in Honeycomb.
# environment variable
HONEYCOMB_TRACES_DATASET=my-traces
# system property
honeycomb.traces.dataset=my-traces
Run your application with the automatic instrumentation agent as a sidecar:
java -javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar -jar /path/to/myapp.jar
You can also include configuration values with an invocation of your application:
java \
-Dhoneycomb.config.file=/path/to/properties/file \
-javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar -jar /path/to/myapp.jar
For more configuration options, read Honeycomb OpenTelemetry Distribution for Java.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
This is the complete list of configuration options for the Honeycomb Agent. It can be configured using system properties or environment variables. System properties take precedence over environment variables if both are specified.
System Property / Environment Variable |
Default Value | Description |
---|---|---|
honeycomb.api.key HONEYCOMB_API_KEY |
None | [required – see note below] Your Honeycomb API key |
service.name SERVICE_NAME |
None | [required – see note below] service.name attribute, where all trace data is sent |
honeycomb.traces.apikey HONEYCOMB_TRACES_APIKEY |
Value of honeycomb.api.key |
Your Honeycomb API key for sending traces |
honeycomb.metrics.apikey HONEYCOMB_METRICS_APIKEY |
Value of honeycomb.api.key |
Your Honeycomb API key for sending metrics |
honeycomb.metrics.dataset HONEYCOMB_METRICS_DATASET |
None | Honeycomb dataset where metrics will be sent (metrics will not be exported if this is not configured) |
honeycomb.api.endpoint HONEYCOMB_API_ENDPOINT |
https://api.honeycomb.io:443 |
Honeycomb ingest endpoint. |
honeycomb.traces.endpoint HONEYCOMB_TRACES_ENDPOINT |
Value of honeycomb.api.endpoint |
Honeycomb ingest endpoint for traces (defaults to the value of HONEYCOMB_API_ENDPOINT ). |
honeycomb.metrics.endpoint HONEYCOMB_METRICS_ENDPOINT |
Value of honeycomb.api.endpoint |
Honeycomb ingest endpoint for metrics (defaults to the value of HONEYCOMB_API_ENDPOINT ). |
sample.rate SAMPLE_RATE |
1 (retain all data) |
Sample rate for the deterministic sampler. Must be a positive integer. |
For all other agent configuration, refer to the OpenTelemetry documentation on the Java Agent Configuration. This contains details about configuring and suppressing specific instrumentations.
By default, the Honeycomb OpenTelemetry SDK uses the gRPC protocol to send telemetry data. To use HTTP instead of gRPC, update the protocol using one of the configuration methods:
-Dotel.exporter.otlp.protocol=http/protobuf
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add manual instrumentation where appropriate. You can use manual instrumentation whether you are using the Agent or the Builder.
To add manual instrumentation, add the Honeycomb OpenTelemetry SDK as a dependency in your project. This provides access to the OpenTelemetry APIs, which lets you access the currently executing span and add attributes to it, and/or to create new spans.
dependencies {
implementation('io.honeycomb:honeycomb-opentelemetry-sdk:1.5.2')
}
<dependency>
<groupId>io.honeycomb</groupId>
<artifactId>honeycomb-opentelemetry-sdk</artifactId>
<version>1.5.2</version>
</dependency>
To create spans, you need to get a Tracer
.
import io.opentelemetry.api;
//...
Tracer tracer = openTelemetry.getTracer("tracer.name.here");
When you create a Tracer
, OpenTelemetry requires you to give it a name as a string.
This string is the only required parameter.
When traces are sent to Honeycomb, the name of the Tracer
is turned into the library.name
field, which can be used to show all spans created from a particular tracer.
In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.
The library.name
field is also used with traces created from instrumentation libraries.
It is often beneficial to add attributes to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your service in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID.
In your code, import io.opentelemetry.api.trace.Span
to get access to the span.
import io.opentelemetry.api.trace.Span;
...
Span span = Span.current();
span.setAttribute("user.id", user.getId());
This will add a user.id
field to the current span so that you can use the field in WHERE
, GROUP BY
or ORDER
clauses in the Honeycomb query builder.
Automatic instrumentation can show the shape of requests to your system, but only you know the really important parts.
In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans.
To do this, select the Tracer
registered by the Agent and build a span.
In your code, import io.opentelemetry.api.GlobalOpenTelemetry
, io.opentelemetry.api.trace.Span
, and io.opentelemetry.api.trace.Tracer
.
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
...
Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");
Span span = tracer.spanBuilder("expensive-query").startSpan();
// ... do cool stuff
span.end();
You can also use the annotation @WithSpan
to wrap the execution of a method with a span.
The span will be automatically closed once the method has completed.
Unless explicitly specified, the span will be named className.methodName
.
To override the name of the span, add a name in parentheses as an argument.
In your code, import io.opentelemetry.instrumentation.annotations.WithSpan
to allow usage of this annotation.
import io.opentelemetry.instrumentation.annotations.WithSpan;
...
@WithSpan("importantSpan")
public String getImportantInfo() {
return importantInfo;
}
Sometimes you will want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.
We will leverage the OpenTelemetry concept of baggage to do this.
Baggage allows you to add a key
with a value
as an attribute to every subsequent child span of the current application context.
In your code, import io.opentelemetry.api.baggage.Baggage
to allow use of the Baggage
class.
import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.context.Scope;
...
try (final Scope ignored = Baggage.current()
.toBuilder()
.put("app.username", name)
.build()
.makeCurrent()
) {
// all subsequently created span in this block will have the `app.username` attribute
}
Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.
The OpenTelemetry documentation for Java has a comprehensive set of topics on manual instrumentation and the Annotations API.
Deterministic head sampling can be used with the Java Agent, with or without manual instrumentation.
The agent will read these variables and expect an integer that represents the sample rate you would like to apply.
For example, a value of 5
means that one out of every five traces will be sent to Honeycomb.
To add sampling to the agent, specify the SAMPLE_RATE
environment variable or the -Dsample.rate
system property when invoking the Java Virtual Machine (JVM), as shown in Advanced Configuration.
The value of your sample rate must be a positive integer.
If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.
The JVM emits metrics at regular intervals that can be sent automatically to a Honeycomb dataset.
These metrics will show up in data as runtime.jvm.gc.count
, runtime.jvm.gc.time
, runtime.jvm.memory.area
, and so on.
If an instrumentation library also supports emitting metrics, then those metrics will be sent as well.
Metrics are currently disabled by default.
To enable metrics, set a HONEYCOMB_METRICS_DATASET
environment variable.
Optionally, set a specific HONEYCOMB_METRICS_APIKEY
or HONEYCOMB_METRICS_ENDPOINT
(if different from the traces API key and endpoint), as shown in Advanced Configuration.
When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.
Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.
Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.
If you do not wish to use the automatic instrumentation agent, and/or wish to do more advanced instrumentation like adding other span processors or extensions, you can use the Honeycomb SDK to instantiate an OpenTelemetryConfiguration
instance, which provides access to a Tracer
and the rest of the OpenTelemetry API.
This should be configured as early as possible in the entry point of your application. Keep in mind, this builder is not required if the agent is in use.
Aside from requiring this builder, setting up manual instrumentation will be the same as when using the agent.
The OpenTelemetryConfiguration
builder provides convenient methods for configuration:
import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.honeycomb.opentelemetry.sdk.trace.samplers.DeterministicTraceSampler; // optional
import io.honeycomb.opentelemetry.sdk.trace.spanprocessors.BaggageSpanProcessor; // optional
public OpenTelemetry honeycomb() {
return OpenTelemetryConfiguration.builder()
.setApiKey("your-api-key")
.setServiceName("your-service-name")
.setOtlpProtocol("http/protobuf") // optional - use HTTP instead of gRPC to transmit telemetry
.setSampler(new DeterministicTraceSampler(2)) // optional - sends 1/2 of all traces generated
.addSpanProcessor(new BaggageSpanProcessor()) // optional
.buildAndRegisterGlobal();
}
When using the Honeycomb SDK builder, specify a span processor and/or a sampler with a sample rate as an argument to the constructor.
The BaggageSpanProcessor
is available as part of the Honeycomb SDK; it is what allows for multi-span attributes.
The DeterministicTraceSampler
is available as part of the Honeycomb SDK that samples based on a common field, given a sample rate.
The setSampler
method accepts any instance of a class that implements the io.opentelemetry.sdk.trace.samplers.Sampler
interface.
Look at the provided deterministic sampler as an example of how to create your own custom sampler, if needed.
Another option for using the agent is to include it when running Maven targets or Gradle tasks. For example, here is a possible configuration to attach the Java agent when running a Spring Boot app:
bootRun {
environment = ["HONEYCOMB_API_KEY": System.getenv("HONEYCOMB_API_KEY"),
"OTEL_SERVICE_NAME": System.getenv("OTEL_SERVICE_NAME")]
jvmArgs = ["-javaagent:honeycomb-opentelemetry-javaagent-1.5.2.jar"]
}
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<environmentVariables>
<OTEL_SERVICE_NAME>${env.OTEL_SERVICE_NAME}</OTEL_SERVICE_NAME>
<HONEYCOMB_API_KEY>${env.HONEYCOMB_API_KEY}</HONEYCOMB_API_KEY>
</environmentVariables>
<agents>
<agent>
honeycomb-opentelemetry-javaagent-1.5.2.jar
</agent>
</agents>
</configuration>
</plugin>
</plugins>
</build>
Be sure to set the environment variables for HONEYCOMB_API_KEY
and OTEL_SERVICE_NAME
as described in the Configure step above.
The primary purpose of Honeycomb’s Distribution for Java is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Java, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.
The Honeycomb Distribution reads specific variables and translates them to variables understood by upstream OpenTelemetry.
For example, when using the distribution, the system property honeycomb.traces.endpoint
is converted to the OpenTelemetry system property otel.exporter.otlp.traces.endpoint
.
Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.
As per the OpenTelemetry specification, it is also required to set a service.name
resource in your SDK configuration.
The service name is used as the name of a dataset to store trace data in Honeycomb.
When using OpenTelemetry for Java, all the following configuration properties are required:
System Property / Environment Variable |
Value |
---|---|
otel.traces.exporter OTEL_TRACES_EXPORTER |
otlp |
otel.metrics.exporter OTEL_METRICS_EXPORTER |
otlp (*) |
otel.exporter.otlp.endpoint OTEL_EXPORTER_OTLP_ENDPOINT |
https://api.honeycomb.io |
otel.exporter.otlp.traces.endpoint OTEL_EXPORTER_OTLP_TRACES_ENDPOINT |
https://api.honeycomb.io/v1/traces (defaults to value of OTEL_EXPORTER_OTLP_ENDPOINT ) |
otel.exporter.otlp.metrics.endpoint OTEL_EXPORTER_OTLP_METRICS_ENDPOINT |
https://api.honeycomb.io/v1/metrics (*) |
otel.exporter.otlp.headers OTEL_EXPORTER_OTLP_HEADERS |
x-honeycomb-team=HONEYCOMB_API_KEY |
otel.exporter.otlp.traces.headers OTEL_EXPORTER_OTLP_TRACES_HEADERS |
x-honeycomb-team=HONEYCOMB_API_KEY (defaults to value of OTEL_EXPORTER_OTLP_HEADERS ) |
otel.exporter.otlp.metrics.headers OTEL_EXPORTER_OTLP_METRICS_HEADERS |
x-honeycomb-team=HONEYCOMB_API_KEY,x-honeycomb-dataset=HONEYCOMB_DATASET (*) |
otel.service.name OTEL_SERVICE_NAME |
service.name attribute to be used for all spans |
Fields marked with an asterisk (*) are required for exporting metrics to Honeycomb.
If you do not have a service name set in TracerProvider configuration code, you must also set the OTEL_SERVICE_NAME
environment variable.
More details on configuration options can be found on GitHub at OpenTelemetry SDK Autoconfigure.
If desired, a configuration file can be used to store environment variables or system properties for the Java Agent.
A path to the configuration file must be provided using either a system property of otel.javaagent.configuration-file
or an environment variable of OTEL_JAVAAGENT_CONFIGURATION_FILE
.
For example, to specify a configuration file using system properties to run the agent with an app:
java -javaagent:path/to/opentelemetry-javaagent.jar \
-Dotel.javaagent.configuration-file=path/to/config-file \
-jar myapp.jar
By default, OpenTelemetry for Java uses gRPC protocol. To use HTTP instead of gRPC, update the protocol using one of the configuration methods:
-Dotel.exporter.otlp.protocol=http/protobuf
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
The protocol can also be set specific to each signal, such as OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
.
Traces and Metrics Endpoint configuration options, as well as OpenTelemetry Headers, are listed in the OpenTelemetry for Java chart if sending data to Honeycomb directly.
If using an OpenTelemetry Collector, specify the endpoint of the collector, and add the headers to the collector configuration file.
You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.
Use a TraceIdRatioBased
sampler, with a ratio expressed as 1/N
.
Then, also create a resource attribute called SampleRate
with the value of N
.
This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.
In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).
System Property / Environment Variable |
Value |
---|---|
otel.traces.sampler OTEL_TRACES_SAMPLER |
traceidratio |
otel.traces.sampler.arg OTEL_TRACES_SAMPLER_ARG |
0.5 |
otel.resource.attributes OTEL_RESOURCE_ATTRIBUTES |
SampleRate=2 |
The value of SampleRate
must be a positive integer.
When using the OTEL_EXPORTER_OTLP_ENDPOINT
environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.
For example, when exporting trace data, the endpoint is updated to append v1/traces
.
When exporting metrics data, the endpoint is updated to append v1/metrics
.
The same modification is not necessary for gRPC.
By setting OTEL_EXPORTER_OTLP_ENDPOINT
to https://api.honeycomb.io
, traces are sent to https://api.honeycomb.io/v1/traces
and metrics to https://api.honeycomb.io/v1/metrics
.
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT
instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT
.
When using a signal-specific environment variable, these paths must be appended manually.
Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
for traces, appending the endpoint with v1/traces
, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
for metrics, appending the endpoint with v1/metrics
.
Send both traces and metrics to Honeycomb using this method by setting the following variables:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics
More details about endpoints and signals can be found in the OpenTelemetry Specification.
The service name is a required configuration value.
If it is unspecified, all trace data will be sent to a default dataset called unknown_service
.
If a This trace has multiple spans sharing the same non-null span ID
error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.
One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.
To enable debugging when running with the OpenTelemetry Java Agent, set the otel.javaagent.debug
system property or OTEL_JAVAAGENT_DEBUG
environment variable to true
.
When this setting is provided, the Agent configures a LoggingSpanExporter that logs traces & metrics data.
If you are not using the OpenTelemetry Java Agent, you can add a LoggingSpanExporter
to your builder configuration.
This will require adding another dependency on io.opentelemetry:opentelemetry-exporter-logging
.
import io.honeycomb.opentelemetry.OpenTelemetryConfiguration;
import io.opentelemetry.exporter.logging.LoggingSpanExporter; // for debugging
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; // for debugging
public OpenTelemetry honeycomb() {
return OpenTelemetryConfiguration.builder()
.setApiKey("your-api-key")
.setServiceName("your-service-name")
.addSpanProcessor(SimpleSpanProcessor.create(new LoggingSpanExporter())) // for debugging
.buildAndRegisterGlobal();
}
Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.
A gRPC transport is required to transmit OpenTelemetry data. HoneycombSDK includes grpc-netty-shaded
.
If you are using another gRPC dependency, version conflicts can come up with an error like this:
java.lang.NoSuchMethodError: io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51).
If you would like to use another gRPC transport, you can exclude the grpc-netty-shaded
transitive dependency:
dependencies {
implementation('io.honeycomb:honeycomb-opentelemetry-sdk:1.5.2') {
exclude group: 'io.grpc', module: 'grpc-netty-shaded'
}
}
<dependency>
<groupId>io.honeycomb</groupId>
<artifactId>honeycomb-opentelemetry-sdk</artifactId>
<version>1.5.2</version>
<exclusions>
<exclusion>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
</exclusion>
</exclusions>
</dependency>
You may receive a 464
error response from the Honeycomb API when sending telemetry using gRPC and HTTP1.
The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.
Additionally, older JVMs may not have sufficient gRPC support and may attempt to send telemetry using HTTP1.
To resolve this, either update to a newer JVM or use http/protobuf
as the transfer protocol.