Honeycomb has an OpenTelemetry Distribution for Node.js to instrument your applications and send trace data to Honeycomb. In addition to simplified OpenTelemetry initialization, the Honeycomb Distribution makes it easier to add multi-span attributes and configure sampling, as well as link directly to your traces in Honeycomb with local visualizations.
The primary purpose of Honeycomb’s Distribution for Node.js is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution uses OpenTelemetry for JavaScript, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution. Refer to how to use OpenTelemetry without the Honeycomb Distribution.
Automatic instrumentation is enabled by adding instrumentation packages. Manual instrumentation can be added using the OpenTelemetry API.
These instructions will explain how to set up automatic and manual instrumentation for a Node.js service. In order to follow along, you will need:
There are several examples that configure applications to send OpenTelemetry data to Honeycomb.
Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry’s automatic instrumentation meta package and Honeycomb’s OpenTelemetry Node.js distribution package:
npm install --save \
@opentelemetry/auto-instrumentations-node \
@honeycombio/opentelemetry-node
Module | Description |
---|---|
auto-instrumentations-node |
OpenTelemetry’s meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like express , dns , http , and more. |
opentelemetry-node |
Honeycomb’s Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. |
Alternatively, install individual instrumentation packages.
Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry’s automatic instrumentation meta package and Honeycomb’s OpenTelemetry Node.js distribution package:
yarn add \
@opentelemetry/auto-instrumentations-node \
@honeycombio/opentelemetry-node
Module | Description |
---|---|
auto-instrumentations-node |
OpenTelemetry’s meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like express , dns , http , and more. |
opentelemetry-node |
Honeycomb’s Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. |
Alternatively, install individual instrumentation packages.
If using TypeScript, install ts-node
to run the code:
yarn add --dev ts-node
Create an initialization file, commonly known as the tracing.ts
file:
// Example filename: tracing.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { HoneycombSDK } from '@honeycombio/opentelemetry-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
// Uses environment variables named HONEYCOMB_API_KEY and OTEL_SERVICE_NAME
const sdk: NodeSDK = new HoneycombSDK({
instrumentations: [
getNodeAutoInstrumentations({
// We recommend disabling fs automatic instrumentation because
// it can be noisy and expensive during startup
'@opentelemetry/instrumentation-fs': {
enabled: false,
},
}),
],
});
sdk.start();
Create an initialization file, commonly known as the tracing.js
file:
// Example filename: tracing.js
'use strict';
const { HoneycombSDK } = require('@honeycombio/opentelemetry-node');
const {
getNodeAutoInstrumentations,
} = require('@opentelemetry/auto-instrumentations-node');
// Uses environment variables named HONEYCOMB_API_KEY and OTEL_SERVICE_NAME
const sdk = new HoneycombSDK({
instrumentations: [getNodeAutoInstrumentations({
// We recommend disabling fs automatic instrumentation because
// it can be noisy and expensive during startup
'@opentelemetry/instrumentation-fs': {
enabled: false,
},
})]
});
sdk.start()
Use environment variables to configure the Honeycomb OpenTelemetry Node.js distribution package:
export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"
Variable | Description |
---|---|
OTEL_SERVICE_NAME |
Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. |
HONEYCOMB_API_KEY |
Your API Key generated in Honeycomb. Learn how to find your Honeycomb API Key. |
Use environment variables to configure the Honeycomb OpenTelemetry Node.js distribution package:
export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"
Variable | Description |
---|---|
OTEL_SERVICE_NAME |
Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. |
HONEYCOMB_API_KEY |
Your API Key generated in Honeycomb. Learn how to find your Honeycomb API Key. |
If you are a Honeycomb Classic user, you must also specify the Dataset using the HONEYCOMB_DATASET
environment variable.
A Dataset is a bucket where data gets stored in Honeycomb.
export HONEYCOMB_DATASET="your-dataset"
To set configuration in code, instead of environment variables, add to the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()]
})
This is the complete list of configuration options for the Honeycomb Distribution.
Environment Variable | Default Value | Description |
---|---|---|
HONEYCOMB_API_KEY |
None | [required – see note below] Your Honeycomb API key |
OTEL_SERVICE_NAME |
unknown_service |
[required – see note below] service.name attribute, where all trace data is sent |
HONEYCOMB_TRACES_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending traces |
HONEYCOMB_METRICS_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending metrics |
HONEYCOMB_METRICS_DATASET |
None | Honeycomb dataset where metrics will be sent |
HONEYCOMB_API_ENDPOINT |
api.honeycomb.io:443 |
Honeycomb ingest endpoint |
HONEYCOMB_TRACES_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for traces (defaults to the value of HONEYCOMB_API_ENDPOINT ) |
HONEYCOMB_METRICS_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for metrics (defaults to the value of HONEYCOMB_API_ENDPOINT ) |
SAMPLE_RATE |
1 (retain all data) |
Sample rate for the deterministic sampler. Must be a positive integer. |
OTEL_METRICS_ENABLED |
false |
Enable metrics export (metrics dataset must be configured as well) |
HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS |
false |
Enable local visualizations |
DEBUG |
false |
Enable debug mode |
Run the Node.js app and include the initialization file you created:
ts-node -r ./tracing.ts YOUR_APPLICATION_NAME.ts
Be sure to replace YOUR_APPLICATION_NAME
with the name of your application’s main file.
Alternatively, you can import the initialization file as the first step in your application lifecycle.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
Run the Node.js app and include the initialization file you created:
node -r ./tracing.js YOUR_APPLICATION_NAME.js
Be sure to replace YOUR_APPLICATION_NAME
with the name of your application’s main file.
Alternatively, you can import the initialization file as the first step in your application lifecycle.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
By default, the Honeycomb OpenTelemetry SDK uses the HTTP protocol (http/protobuf
) to send telemetry data.
To use gRPC instead of HTTP/protobuf, update the protocol to grpc:
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
Alternatively, if you prefer to configure this protocol in code, you can configure it with the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
protocol: OtlpProtocolKind.Grpc,
})
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
protocol: "grpc",
})
Automatic instrumentation is the easiest way to get started with instrumenting your code, but in order to get the most insight into your system, you should add manual instrumentation where appropriate. To do this, use the OpenTelemetry API to access the currently executing span and add attributes to it, and/or to create new spans.
Adding manual instrumentation requires the the OpenTelemetry API package. If not installed yet, use this command for installation:
npm install --save @opentelemetry/api
yarn add @opentelemetry/api
It is often beneficial to add attributes to a currently executing span in a trace. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your service in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID:
import opentelemetry from '@opentelemetry/api';
function handleUser(user: User) {
let activeSpan = opentelemetry.trace.getActiveSpan();
activeSpan.setAttribute("user.id", user.getId());
}
const opentelemetry = require("@opentelemetry/api");
function handleUser(user) {
let activeSpan = opentelemetry.trace.getActiveSpan();
activeSpan.setAttribute("user.id", user.getId());
}
This will add a user.id
field to the current span so that you can use the field in WHERE
, GROUP BY
or ORDER
clauses in the Honeycomb query builder.
To start manually tracing, you must initialize a tracer.
import opentelemetry from '@opentelemetry/api';
const tracer = opentelemetry.trace.getTracer("tracer.name.here");
const opentelemetry = require("@opentelemetry/api");
const tracer = opentelemetry.trace.getTracer("tracer.name.here");
When you create a Tracer
, OpenTelemetry requires you to give it a name as a string.
This string is the only required parameter.
When traces are sent to Honeycomb, the name of the Tracer
is turned into the library.name
field, which can be used to show all spans created from a particular tracer.
In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.
The library.name
field is also used with traces created from instrumentation libraries.
You can then use this tracer to create custom spans.
Automatic instrumentation can show the shape of requests to your system, but only you know the truly important parts. In order to get the full picture of what is happening, you will have to add manual instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API:
import opentelemetry from '@opentelemetry/api';
const tracer = opentelemetry.trace.getTracer("my-service-tracer");
function runQuery() {
tracer.startActiveSpan("expensive-query", (span) => {
// ... do cool stuff
span.end();
});
}
const opentelemetry = require("@opentelemetry/api");
const tracer = opentelemetry.trace.getTracer("my-service-tracer");
function runQuery() {
tracer.startActiveSpan("expensive-query", (span) => {
// ... do cool stuff
span.end();
});
}
Sometimes you want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.
We will leverage the OpenTelemetry concept of baggage to add this attribute.
Baggage allows you to add a key
with a value
as an attribute to every subsequent child span within the current application context.
import {
Context,
context,
propagation,
} from '@opentelemetry/api';
tracer.startActiveSpan('main', (span) => {
span.setAttribute('app.username', name); // add to current span
// new context based on current, with key/values added to baggage
const ctx: Context = propagation.setBaggage(
context.active(),
propagation.createBaggage({ 'app.username': { value: name } })
);
// within the new context, do some work and baggage will be
// applied as attributes on child spans
context.with(ctx, () => {
tracer.startActiveSpan('childSpan', (childSpan) => {
doTheWork();
childSpan.end();
});
});
span.end();
});
tracer.startActiveSpan('main', (span) => {
span.setAttribute('app.username', name); // add to current span
// new context based on current, with key/values added to baggage
const ctx = propagation.setBaggage(
context.active(),
propagation.createBaggage({ 'app.username': { value: name } })
);
// within the new context, do some work and baggage will be
// applied as attributes on child spans
context.with(ctx, () => {
tracer.startActiveSpan('childSpan', (childSpan) => {
doTheWork();
childSpan.end();
});
});
span.end();
});
The OpenTelemetry documentation for JavaScript has a comprehensive set of topics on manual instrumentation.
You can send data to an OpenTelemetry Collector instance or another endpoint by specifying a different endpoint:
export HONEYCOMB_API_ENDPOINT="http://<your-collector-endpoint>"
Alternatively, if you prefer to configure this in code, you can configure your endpoint with the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
endpoint: "http://<your-collector-endpoint>",
})
The Honeycomb SDK will automatically append the endpoint with the appropriate traces-specific path of v1/traces
.
Deterministic head sampling can be used with the Honeycomb SDK, with or without manual instrumentation.
The SDK will read these variables and expect an integer that represents the sample rate you would like to apply.
For example, a value of 5
means that one out of every five traces will be sent to Honeycomb.
To add sampling to the SDK, set the SAMPLE_RATE
environment variable:
export SAMPLE_RATE=5
To set the sample rate in code instead of an environment variable, add sampleRate
to the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
sampleRate: 5,
})
The value of your sample rate must be a positive integer.
If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.
When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.
Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.
Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.
Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.
Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.
By default, the Honeycomb’s OpenTelemetry Distribution for Node.js uses the W3C trace context format.
If you opt to use a different trace context specification than W3C, ensure that both the sending and receiving service are using the same propagation format, and that both services are configured to send data to the same Honeycomb environment.
Honeycomb’s OpenTelemetry Distribution for Node.js can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation.
To enable local visualizations, set the HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS
environment variable to true
:
export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true
Then, run your application:
node -r ./tracing.js APPLICATION_MAIN_FILE.js
To set the local visualizations option in code instead of an environment variable, add localVisualizations
to the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
localVisualizations: true,
})
The output displays the name of the root span and a link to Honeycomb that shows its trace. For example:
Trace for root-span-name
Honeycomb link: <link to Honeycomb trace>
Select the link to view the trace in detail within the Honeycomb UI.
The Honeycomb Distribution reads specific variables and translates them to variables understood by the upstream OpenTelemetry SDK.
For example, the distribution automatically configures exporters to send telemetry data to Honeycomb’s API api.honeycomb.io
.
Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.
Follow the directions below to instrument with OpenTelemetry for JavaScript using the Node SDK.
npm install --save \
@opentelemetry/sdk-node \
@opentelemetry/exporter-trace-otlp-proto \
@opentelemetry/auto-instrumentations-node
Create an initialization file, commonly known as the tracing.js
file, and import it as the first step in your application lifecycle.
Alternatively, include this initialization file with the -r
or --require
Node.js CLI option.
// tracing.js
'use strict';
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-proto");
// The Trace Exporter exports the data to Honeycomb and uses
// environment variables for endpoint, service name, and API Key.
const traceExporter = new OTLPTraceExporter();
const sdk = new NodeSDK({
traceExporter,
instrumentations: [getNodeAutoInstrumentations()]
});
sdk.start()
Configure OpenTelemetry to send events to Honeycomb using environment variables.
The header x-honeycomb-team
is your API key.
Your service name will be used as the Service Dataset in Honeycomb, which is where data is stored.
The service name is specified by OTEL_SERVICE_NAME
.
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
export OTEL_SERVICE_NAME="your-service-name"
Run the Node.js app with the OpenTelemetry initialization file to generate data:
node -r ./tracing.js APPLICATION_MAIN_FILE.js
If you are a Honeycomb Classic user, the Dataset also must be specified using the x-honeycomb-dataset
header.
A Dataset is a bucket where data gets stored in Honeycomb.
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset"
You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.
Use a TraceIdRatioBased
sampler, with a ratio expressed as 1/N
.
Then, also create a resource attribute called SampleRate
with the value of N
.
This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.
In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).
To get access to a sampler, install the core OpenTelemetry package:
npm install --save @opentelemetry/core
Import the TraceIdRatioBasedSampler
, and add as a sampler to the NodeSDK
along with the SampleRate
in the Resource
.
// tracing.js
const { TraceIdRatioBasedSampler } = require("@opentelemetry/core");
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: "<YOUR_SERVICE_NAME>",
SampleRate: 2,
}),
traceExporter,
instrumentations: [getNodeAutoInstrumentations()],
sampler: new TraceIdRatioBasedSampler(0.5),
});
When using the OTEL_EXPORTER_OTLP_ENDPOINT
environment variable with an SDK and an HTTP exporter, the final path of the endpoint is actually modified by the SDK to represent the specific signal being sent.
For example, when exporting trace data, the endpoint is updated to append v1/traces
.
When exporting metrics data, the endpoint is updated to append v1/metrics
.
The same modification is not necessary for gRPC.
By setting OTEL_EXPORTER_OTLP_ENDPOINT
to https://api.honeycomb.io
, traces are sent to https://api.honeycomb.io/v1/traces
and metrics to https://api.honeycomb.io/v1/metrics
.
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
If the desired outcome is to send data to a different endpoint depending on the signal, use OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT
instead of the more generic OTEL_EXPORTER_OTLP_ENDPOINT
.
When using a signal-specific environment variable, these paths must be appended manually.
Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
for traces, appending the endpoint with v1/traces
, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
for metrics, appending the endpoint with v1/metrics
.
Send both traces and metrics to Honeycomb using this method by setting the following variables:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics
More details about endpoints and signals can be found in the OpenTelemetry Specification.
To configure the endpoint URL and API key in code instead of with environment variables, specify /v1/traces
like so:
// Configure OTLPTraceExporter
const traceExporter =
new OTLPTraceExporter({
url: "https://api.honeycomb.io/v1/traces",
headers: {
"x-honeycomb-team": 'your-api-key',
},
});
If a This trace has multiple spans sharing the same non-null span ID
error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.
One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.
To enable debugging when running the Honeycomb OpenTelemetry Node SDK, set the DEBUG
environment variable to true
:
export DEBUG=true
To set the debug option in code instead of an environment variable, add debug
to the HoneycombSDK
:
const sdk = new HoneycombSDK({
apiKey: "your-api-key",
serviceName: "your-service-name",
instrumentations: [getNodeAutoInstrumentations()],
sampleRate: 5,
localVisualizations: true,
debug: true,
})
When the debug setting is enabled, the Honeycomb SDK configures a DiagConsoleLogger that logs telemetry to the console with the log level of Debug.
The debug setting in the Honeycomb SDK will also output to the console the full options configuration, including but not limited to protocol, API Key, and endpoint.
If you are not using the Honeycomb SDK, or if you wish to change the logging level, you can still use the logger directly:
const opentelemetry = require("@opentelemetry/api");
opentelemetry.diag.setLogger(
new opentelemetry.DiagConsoleLogger(),
opentelemetry.DiagLogLevel.DEBUG
);
Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.
Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP protobuf definitions are 0.7.0 for traces and metrics.
If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.
You may receive a 464
error response from the Honeycomb API when sending telemetry using gRPC and HTTP1.
The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.
.proto
Files Error If your application builds using a bundler, like Webpack or ESBuild, or if your application uses TypeScript, you will see an error that says that a specific .proto
file is not found:
opentelemetry/proto/collector/trace/v1/trace_service.proto not found in any of the include paths <directory>/protos
This error appears because the .proto
files are not imported or required by the library that uses them, so bundlers do not know to include them in the final build.
There are two ways to work around this error:
/protos
Directory to the Correct Location Modify the bundler configuration to copy the /protos
directory from the library to the same level as the build directory:
const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
... other config ...
plugins: [
new CopyPlugin({
patterns: [
{
from: "./node_modules/@opentelemetry/otlp-grpc-exporter-base/build/protos/**/*",
to: "./protos"
}
],
}),
],
};
const { copy } = require('esbuild-plugin-copy')
require('esbuild').build({
... other config ...
plugins: [
copy({
resolveFrom: 'cwd',
assets: {
from: ['./node_modules/@opentelemetry/otlp-grpc-exporter-base/build/protos/**/*'],
to: ['./protos'],
keepStructure: true
},
}),
],
}).catch(() => process.exit(1));
TypeScript configuration currently cannot specify which extra files to copy for your build.
In this case, use a postbuild
npm script to copy the proto files to the correct location.