The Node.js Beeline provides instant, per-request visibility for your Express application. It automatically instruments many common Node.js packages with traces and events to capture useful information from your application.
The Node.js Beeline allows you to slice and dice requests by endpoint, status, or even User ID, with zero custom instrumentation required. It creates traces automatically out of the box. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey, you may find you’d like to add new events or traces to add more details specific to your app. The Node.js Beeline provides simple interfaces for adding both.
If you’d like to see more options in the Node.js Beeline, please file an issue or vote up one already filed! You can also contact us at support@honeycomb.io.
If you prefer more control over your application’s instrumentation, the Node.js Beeline has an API of its own for adding to traces.
You can find your API key on your Team Settings page. If you don’t have a API key yet, sign up for a Honeycomb trial.
If you’ve got a NodeJS express or fastify app, you can get request-level instrumentation from those frameworks and other supported packages you use, magically.
Install the Node.js Beeline package using npm:
npm install --save honeycomb-beeline
Add the following code to the top of your app.js.
Important: It must be before any require or import statements for other packages.
require("honeycomb-beeline")({
writeKey: "YOUR_API_KEY",
dataset: "my-node-distributed-app",
serviceName: "my-node-service"
/* ... additional optional configuration ... */
});
The packaged instrumentations send context to Honeycomb about requests and queries, but they can’t automatically capture all context that you might want. Additional fields are an excellent way to add detail to your events. Try putting a timer around a section of code, add adding per- user information, or details about what it took to craft a response. You can add fields when and where you need to, or for some events but not others. (Error handlers are a good example of this.)
Here is an example of adding a custom field to the currently-active span:
const beeline = require("honeycomb-beeline")();
const calcBigNum = num => {
beeline.addContext({ oldNum: num });
// ... do thing with num
beeline.addContext({ newNum: num });
};
Note that beeline.addContext only adds fields to the current span.
In order to add fields to all downstream spans, use beeline.customContext.add instead:
const beeline = require("honeycomb-beeline")();
const handleInput = () => {
beeline.customContext.add("userId", userId);
};
Note that beeline.customContext.add will prefix your field name with app.,
so that all downstream spans will be populated with an app.userId field.
These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per- user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.
It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.
You can add custom context several ways, including with callbacks or promises. See Custom Context Propagation for more detail on these methods.
We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.
beeline.startTrace() starts a new local trace and initializes the async context
propagation machinery. You must have an active trace for the tracing API to work
correctly. If you call startSpan when you aren’t currently in a trace, an Error
will be thrown. The instrumentations (which must operate in both trace/non-trace
environments) handle this by checking beeline.traceActive() and only creating
spans if they’re within a trace.
This method also creates a root span for the trace (using beeline.startSpan below),
and adds metadataContext as its initial context. This root span is installed as the current span.
Note: You must call startTrace outside of an async function. Other beeline calls will work inside async functions.
Below is an example of starting a trace:
let trace = beeline.startTrace({
field1: value1,
field2: value2
});
To start a new span in the existing trace, call beeline.startSpan();. This returns a
reference to the span which can then be used in finishSpan like this:
let span = beeline.startSpan({
field1: value1,
field2: value2
});
fs.writeFile(filePath, fileContents, err => {
beeline.customContext.add("fileError", err.toString());
beeline.finishSpan(span);
});
If you’re doing something synchronously (for example, looping, or using a synchronous
node api) you can use withSpan to wrap this operation. Since it returns the return value
of fn, it can be used in an expression context.
Here’s an example:
let sum = beeline.withSpan(
{
task: "calculating the sum"
},
() => {
let s = 0;
for (let i of bigArray) {
s += i;
}
return s;
}
);
After you’ve added all the spans you need to your trace, call beeline.finishTrace(trace);
to send the trace’s root span, and complete the necessary teardown. Below is an example of finishing a trace:
beeline.finishTrace(trace);
Some traces can be expressed as a single function, for example, if you’re doing something synchronously (maybe in a script). In this case you can use, withTrace() as seen below:
beeline.withTrace(
{
task: "writing a file",
filePath
},
() => fs.writeFileSync(filePath, fileContents)
);
// Another example of withTrace, in an expression context:
console.log(
`the answer is ${beeline.withTrace(
{
task: "computing fibonacci number",
n
},
() => computeFib(n)
)}`
);
The following is a list of packages we’ve added instrumentation for. Some add context to events, while others propagate context so the Beeline can create events in downstream packages.
The source for each individual instrumented package can be found in the lib/instrumentation folder on GitHub.
(if you’d like to see anything more here, please file an issue or 👍 one already filed!)
bluebird 🔗 Instrumented only for context propagation
mpromise 🔗 Instrumented only for context propagation
express 🔗 Adds columns with prefix request.
Configuration Options:
| Name | Type |
|---|---|
express.userContext |
Array<string>|Function<(request) => Object> |
express.userContext
If the value of this option is an array, it’s assumed to be an array of string field names of req.user. If a request has req.user, the named fields are extracted and added to events with column names of express.user.$fieldName.
For example:
If req.user is an object { id: 1, username: "toshok" } and your config settings include express: { userContext: ["username"] }, the following will be included in the express event sent to honeycomb:
request.user.username |
|---|
toshok |
If the value of this option is a function, it will be called on every request and passed the request as the sole argument. All key-values in the returned object will be added to the event. If the function returns a falsey value, no columns will be added. To replicate the above Array-based behavior, you could use the following config: express: { userContext: (req) => req.user && { username: req.user.username } }
This function isn’t limited to using the request object, and can pull info from anywhere to enrich the data sent about the user.
http 🔗 Adds columns with prefix http.
https 🔗 Adds columns with prefix https.
mysql2 🔗 Adds columns with prefix db.
pg 🔗 Adds columns with prefix db.
sequelize 🔗 Instrumented only for context propagation
mongoose 🔗 Instrumented only for context propagation
mongodb 🔗 Adds columns with prefix db.
Configuration options:
| Name | Type |
|---|---|
mongodb.includeDocuments |
boolean |
mongodb.includeDocuments
If true, documents in the api will be JSON serialized and included in the events sent to honeycomb.
react-dom/server 🔗 Adds columns with prefix react.
child_process 🔗 Instrumented only for context propagation
The additional optional configuration in the code example above is where
you can add global settings (Honeycomb credentials and dataset name) or
per-instrumentation settings:
{
writeKey: "YOUR_API_KEY",
dataset: "my-dataset-name"
$instrumentationName: {
/* instrumentation specific settings */
}
}
dataset is optional. If you do not specify a dataset, it will default to nodejs.
You may also specify writeKey and dataset by setting HONEYCOMB_WRITEKEY and
HONEYCOMB_DATASET in your environment.
To add custom instrumentation settings, specify them in your config object as a
key/value pair using the name of the instrumentation as the key. For example, to
add configuration options for express, your config object might look like:
{
writeKey: "YOUR_API_KEY",
dataset: "my-dataset-name",
express: {
/* express-specific settings */
}
}
See the Instrumented packages section below for available configuration options for each package.
If you want to disable automatic instrumentation for whatever reason, for either an individual package or all packages, you can pass enabledInstrumentations when configuring the beeline. It should be an array of package names to automatically instrument. For instance, if you want to enable the beeline instrumentation only for the http and https packages:
require("honeycomb-beeline")({
enabledInstrumentations: ["http", "https"]
/* ... additional configuration ... */
});
The beeline also exports getInstrumentations, which returns a list of all enabled instrumentation. You can use this to filter out specific instrumentations you want to disable. If you want to enable all instrumentation except mongodb:
const beeline = require("honeycomb-beeline");
beeline({
enabledInstrumentations: beeline
.getInstrumentations()
.filter(i => i !== "mongodb")
/* ... additional configuration ... */
});
Finally, to disable all automatic instrumentation, pass an empty array as in:
require("honeycomb-beeline")({
enabledInstrumentations: []
/* ... additional configuration ... */
});
If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the presendHook is your opportunity to make these changes. Examples are scrubbing sensitive data (eg you may want to ensure specific fields are dropped or hashed) or augmenting data (eg making out-of-band translations between an ID and a more human readable version). Similar to the samplerHook discussed below, you pass the presendHook a function that will be called on every span with the span as an argument. The function is free to mutate the span passed in and those mutations will be what finally gets sent to Honeycomb.
As an example, say we have some HTTP requests that provide sensitive values which have been added to a span. This code will examine all spans just before they’re sent to Honeycomb and remove the sensitive values.
const beeline = require("honeycomb-beeline");
beeline({
presendHook: function(ev) {
// either:
ev.data.scrubMe = undefined;
// or:
// delete ev.data.scrubMe
}
});
beeline.finishTrace(beeline.startTrace({ name: "foo", scrubMe: "sensitive data" }));
If the environment variables HTTPS_PROXY or https_proxy are set, the Beeline will pick up and configure itself to use the proxy for all event traffic to Honeycomb.
We have built-in support for sampling in the Beeline. Simply set the sampleRate variable to your beeline configuration. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events:
require("honeycomb-beeline")({
writeKey: "YOUR_API_KEY",
dataset: "my-dataset-name",
sampleRate: 5
/* ... additional optional configuration ... */
});
Sampling by default will apply the same sampling decision to all spans in a trace, so adding sampling won’t break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent. If sampling across multiple Beeline-instrumented services, set the same sample rate in all beeline configurations to avoid breaking traces.
The samplerHook configuration option is available to customize the logic used for deterministic sampling. To avoid breaking traces with the custom samplerHook option, ensure that sampling logic is applied to data that all spans within a trace will have (such as trace.trace_id). If needed, you may promote span fields to trace-level (making them available on all spans within a trace) by calling addTraceContext() at the beginning of a trace, for instance: beeline.addTraceContext({ request_route: '/x/alive' }).
A custom samplerHook must return an object with this structure:
{
shouldSample: boolean, // false will drop the event, true will keep it
sampleRate: number // optional, will be reported to honeycomb
}
For example, assume you have instrumented an HTTP server. You’d like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so:
const beeline = require("honeycomb-beeline");
const createHash = require("crypto").createHash;
beeline({
writeKey: "YOUR_API_KEY",
dataset: "my-dataset-name",
samplerHook: samplerHook,
});
// deterministicSampler function based on https://github.com/honeycombio/beeline-nodejs/blob/main/lib/deterministic_sampler.js
function deterministicSampler(traceId, sampleRate) {
const MAX_UINT32 = Math.pow(2, 32) - 1;
const sum = createHash("sha1")
.update(traceId)
.digest();
const upperBound = (MAX_UINT32 / sampleRate) >>> 0;
return sum.readUInt32BE(0) <= upperBound;
}
function samplerHook(data) {
// default sample rate to 10
let sampleRate = 10;
// default sampling decision to deterministic based on trace_id
let shouldSample = deterministicSampler(data["trace.trace_id"], sampleRate);
// drop all health checks requests
if (data["request.path"] === "/x/alive") {
shouldSample = false;
sampleRate = 0;
}
return {
shouldSample,
sampleRate
}
}
app.get('/x/alive', function(req, res) {
res.send("I'm alive");
});
Note: Defining a sampling hook overrides the default deterministic sampling behavior for trace IDs. For head-based sampling behavior across a more complicated trace than the health check example above, you must ensure sampling decisions are based on a value all spans will have (such as trace_id or a custom trace context as we did above). Otherwise, you will get incomplete traces.
If a single trace in your system can traverse multiple processes, you’ll need a way to connect the spans emitted by each service into a single trace. This is handled by propagating trace context via an HTTP header.
The Honeycomb beelines support trace headers in a Honeycomb specific format as well as the W3C Trace Context format.
Auto-instrumentation supports trace propagation automatically, as long as your services are using
the Honeycomb beeline, and an instrumented component to send and receive requests (e.g. express and https).
In order to support distributed traces that include services instrumented with a Honeycomb Beeline and OpenTelemetry, the Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers, the format used by OpenTelemetry.
In order to specify that a service should parse W3C Trace Context headers from
incoming requests, you must specify an httpTraceParserHook in the beeline configuration.
An httpTraceParserHook is a function that takes an HTTP request as an argument and
returns a Honeycomb trace context object. The HTTP request is provided to the function
so that the author can make decisions about whether to trust the incoming headers based
on information contained in the request (e.g. perhaps you don’t want to accept headers
from the public internet).
const beeline = require('honeycomb-beeline');
beeline({
writeKey: 'YOUR_API_KEY',
dataset: 'my-dataset',
httpTraceParserHook: beeline.w3c.httpTraceParserHook
});
In order to send trace propagation headers in a supported format, you must specify an
httpTracePropagationHook in the beeline configuration.
An httpTracePropagationHook is a function that takes a Honeycomb trace context object
as an argument and returns a map of name, value pairs representing serialized headers.
const beeline = require('honeycomb-beeline');
beeline({
writeKey: 'YOUR_API_KEY',
dataset: 'my-dataset',
httpTracePropagationHook: beeline.w3c.httpTracePropagationHook
});
Because we specified an httpTracePropagationHook that returns a serialized header in W3C trace context format,
the outgoing request will include the appropriate trace context header.
There are two general approaches to finding out what’s wrong when the Node.js Beeline isn’t doing what you expect.
Enable debug output (sent to STDOUT) by setting DEBUG=honeycomb-beeline:*
in your environment. This will print the JSON representation
of events to the terminal instead of sending them to Honeycomb.
This lets you quickly see what’s getting sent
and allows you to modify your code accordingly.
$ DEBUG=honeycomb-beeline:*
There can be a number of reasons for this, however, there is a known issue with the Node.js Beeline, and
the auto-instrumented AWS trace header propagation. If you are using AWS, and have not instrumented it to send
events to Honeycomb (e.g. using the Honeycomb AWS Bundle), you may
notice missing root spans. To override the default AWS trace header propagation behavior, you can configure the
Beeline to use an httpTraceParserHook:
const beeline = require("honeycomb-beeline");
beeline({
writeKey: "YOUR_API_KEY",
dataset: "my-dataset-name",
httpTraceParserHook: beeline.honeycomb.httpTraceParserHook,
});
The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See Distributed Trace Propagation for more details.
Here is a sample event created by the Node.js Beeline:
{
"Timestamp": "2018-03-20T00:47:25.339Z",
"request.base_url": "",
"request.fresh": false,
"request.host": "localhost",
"request.http_version": "HTTP/1.1",
"request.remote_addr": "127.0.0.1",
"request.method": "POST",
"request.original_url": "/checkValid",
"request.path": "/checkValid",
"request.scheme": "http",
"request.query": "{}",
"request.secure": false,
"request.url": "/checkValid",
"request.xhr": true,
"response.status_code": "200",
"meta.instrumentation_count": 4,
"meta.instrumentations": "[\"child_process\",\"express\",\"http\",\"https\"]",
"meta.type": "express",
"meta.version": "4.16.3",
"meta.beeline_version": "1.0.2",
"meta.node_version": "v9.10.0",
"totals.mysql2.count": 2,
"totals.mysql2.duration_ms": 13.291,
"totals.mysql2.query.count": 2,
"totals.mysql2.query.duration_ms": 13.291,
"trace.trace_id": "11ad83a2-ca8d-4918-9db2-27524456d9f7",
"trace.span_id": "4a3892ba-0936-46e1-8e17-31b887326027",
"name": "request",
"service_name": "express",
"duration_ms": 15.229326
}
Here are some examples to get you started querying your app’s behavior:
urlP99 of duration_ms valuesmeta.type == expressP99(duration_ms) in descending (DESC) order
meta.typeP99 of duration_ms valuesP99(duration_ms) in descending (DESC) order
user.emailCOUNTurl == your endpoint url
urlP99 of duration_ms valuesmeta.type == express and xhr == trueP99(duration_ms) in descending (DESC) order
Features, bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!
All contributions will be released under the Apache License 2.0.