1. Docs
2. Troubleshoot
3. Recommended Migrations
4. From Beelines to OpenTelemetry
5. Beeline for Node.js

Beeline for Node.js Reference

Table of Contents

The Node.js Beeline provides instant, per-request visibility for your Express application. It automatically instruments many common Node.js packages with traces and events to capture useful information from your application.

The Node.js Beeline allows you to slice and dice requests by endpoint, status, or even User ID, with zero custom instrumentation required. It creates traces automatically out of the box. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey, you may find you would like to add new events or traces to add more details specific to your app. The Node.js Beeline provides simple interfaces for adding both.

If you would like to see more options in the Node.js Beeline, please file an issue or vote up an already filed one!

If you prefer more control over your application’s instrumentation, the Node.js Beeline has an API of its own for adding to traces.

Requirements 

• Node.js 10+
• A Honeycomb API key

You can find your API key in your Environment Settings. If you do not have a API key yet, sign up for a free Honeycomb account.

Quick Installation 

Adding Context to Events 

The packaged instrumentations send context to Honeycomb about requests and queries, but they can not automatically capture all context that you might want. Additional fields are an excellent way to add detail to your events. Try putting a timer around a section of code, add adding per-user information, or details about what it took to craft a response. You can add fields when and where you need to, or for some events but not others. (Error handlers are a good example of this.)

Here is an example of adding a custom field to the currently-active span:

const beeline = require("honeycomb-beeline")();

const calcBigNum = num => {
  beeline.addContext({ oldNum: num });
  // ... do thing with num
  beeline.addContext({ newNum: num });
};

Note that beeline.addContext only adds fields to the current span. In order to add fields to all downstream spans, use beeline.addTraceContext instead:

const beeline = require("honeycomb-beeline")();

const handleInput = () => {
  beeline.addTraceContext({ userId: userId });
};

Note that beeline.addTraceContext will prefix your field name with app., so that all downstream spans will be populated with an app.userId field.

These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per-user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.

It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.

See the Adding Context to Spans section for more information on the full API available to you.

Adding Spans To a Trace Or Starting New Traces 

We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.

Express and Fastify instrumentations automatically start and finish traces for incoming requests. If you are using a different framework, choose to disable those automatic instrumentations. If you have code that runs outside the request handlers, you will need to manage the trace lifecycle manually.

beeline.startTrace() starts a new local trace and initializes the async context propagation machinery. You must have an active trace for the tracing API to work correctly. If you call startSpan when you are not currently in a trace, an Error will be thrown. The instrumentations, which must operate in both trace/non-trace environments, handle this by checking beeline.traceActive() and only creating spans if they’re within a trace.

This method also creates a root span for the trace (using beeline.startSpan below), and adds metadataContext as its initial context. This root span is installed as the current span.

Below is an example of starting a trace:

let trace = beeline.startTrace({
  field1: value1,
  field2: value2
});

To start a new span in the existing trace, call beeline.startSpan();. This returns a reference to the span which can then be used in finishSpan like this:

let span = beeline.startSpan({
  field1: value1,
  field2: value2
});
let value = calculateSomeValue();
beeline.addTraceContext({
  field3: value
})
beeline.finishSpan(span);

If you are doing something synchronously (for example, looping, or using a synchronous node API) you can use withSpan to wrap this operation. Since it returns the return value of fn, it can be used in an expression context.

Here is an example:

let sum = beeline.withSpan(
  {
    task: "calculating the sum"
  },
  () => {
    let s = 0;
    for (let i of bigArray) {
      s += i;
    }
    return s;
  }
);

After you have added all the spans you need to your trace, call beeline.finishTrace(trace); to send the trace’s root span, and complete the necessary teardown. Below is an example of finishing a trace:

beeline.finishTrace(trace);

Some traces can be expressed as a single function, for example, if you are doing something synchronously (maybe in a script). In this case you can use, withTrace() as seen below:

beeline.withTrace(
  {
    task: "writing a file",
    filePath
  },
  () => fs.writeFileSync(filePath, fileContents)
);

// Another example of withTrace, in an expression context:
console.log(
  `the answer is ${beeline.withTrace(
    {
      task: "computing fibonacci number",
      n
    },
    () => computeFib(n)
  )}`
);

Asynchronous Spans 

Each span in a trace, except the first (root) span, has a parent. Every span must finish before its parent, otherwise you will see incomplete or broken traces in Honeycomb. If you have asynchronous code, where a span might finish after its parent, use beeline.startAsyncSpan() to address this situation. This function takes a callback, which will be called with the new span. When using beeline.startAsyncSpan(), make sure to finish the new span within the callback. For example:

beeline.startAsyncSpan({
  task: "writing a file",
  filePath,
}, span => {
  fs.writeFile(filePath, fileContents, err => {
    if (err) {
      beeline.addTraceContext({ fileError: err.toString() });
    }
    beeline.finishSpan(span);
  });
});

When using async/await syntax, for example:

await beeline.startAsyncSpan({
  task: "writing a file",
  filePath,
}, async (span) => {
  try {
    await fs.writeFile(filePath, fileContents);
  } catch (e) {
    beeline.addTraceContext({ fileError: err.toString() });
  } finally {
    beeline.finishSpan(span);
  }
});

Instrumented Packages 

The following is a list of packages for which we have added instrumentation. Some add context to events, while others propagate context so the Beeline can create events in downstream packages.

The source for each individual instrumented package can be found in the lib/instrumentation folder on GitHub.

(If you would like to see anything more here, please file an issue or 👍 an already filed one!)

bluebird 

Instrumented only for context propagation

mpromise 

Instrumented only for context propagation

express 

Adds columns with prefix request.

Configuration Options:

express.userContext

If the value of this option is an array, it is assumed to be an array of string field names of req.user. If a request has req.user, the named fields are extracted and added to events with column names of express.user.$fieldName.

For example:

If req.user is an object { id: 1, username: "toshok" } and your config settings include express: { userContext: ["username"] }, the following will be included in the express event sent to honeycomb:

If the value of this option is a function, it will be called on every request and passed the request as the sole argument. All key-values in the returned object will be added to the event. If the function returns a falsy value, no columns will be added. To replicate the above Array-based behavior, you could use the following configuration: express: { userContext: (req) => req.user && { username: req.user.username } }

This function is not limited to using the request object, and can pull info from anywhere to enrich the data sent about the user.

http 

Adds columns with prefix http.

https 

Adds columns with prefix https.

mysql2 

Adds columns with prefix db.

pg 

Adds columns with prefix db.

sequelize 

Instrumented only for context propagation

mongoose 

Instrumented only for context propagation

mongodb 

Adds columns with prefix db.

Configuration options:

mongodb.includeDocuments

If true, documents in the api will be JSON serialized and included in the events sent to honeycomb.

react-dom/server 

Adds columns with prefix react.

child_process 

Instrumented only for context propagation

Optional Configuration 

The additional optional configuration in the code example above is where you can add global settings (Honeycomb credentials and service name) or per-instrumentation settings:

{
    writeKey:    "YOUR_API_KEY",
    serviceName: "your-service-name"
    $instrumentationName: {
        /* instrumentation specific settings */
    }
}

You may also specify writeKey and serviceName by setting HONEYCOMB_WRITEKEY and SERVICE_NAME in your environment.

To add custom instrumentation settings, specify them in your configuration object as a key/value pair using the name of the instrumentation as the key. For example, to add configuration options for express, your configuration object might look like:

{
    writeKey: "YOUR_API_KEY",
    serviceName: "your-service-name"
    express: {
    /* express-specific settings */
    }
}

See the Instrumented packages section below for available configuration options for each package.

Customizing Instrumented Packages 

If you want to disable automatic instrumentation for whatever reason, for either an individual package or all packages, you can pass enabledInstrumentations when configuring the Beeline. It should be an array of package names to automatically instrument. For instance, if you want to enable the beeline instrumentation only for the http and https packages:

require("honeycomb-beeline")({
  enabledInstrumentations: ["http", "https"]
  /* ... additional configuration ... */
});

The Beeline also exports getInstrumentations, which returns a list of all enabled instrumentation. You can use this to filter out specific instrumentations you want to disable. If you want to enable all instrumentation except mongodb:

const beeline = require("honeycomb-beeline");
beeline({
  enabledInstrumentations: beeline
    .getInstrumentations()
    .filter(i => i !== "mongodb")
  /* ... additional configuration ... */
});

Finally, to disable all automatic instrumentation, pass an empty array as in:

require("honeycomb-beeline")({
  enabledInstrumentations: []
  /* ... additional configuration ... */
});

Augmenting Or Scrubbing Spans 

If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the presendHook is your opportunity to make these changes. Examples are scrubbing sensitive data (for example, you may want to ensure specific fields are dropped or hashed) or augmenting data (like making out-of-band translations between an ID and a more human readable version). Similar to the samplerHook discussed below, you pass the presendHook a function that will be called on every span with the span as an argument. The function is free to mutate the span passed in and those mutations will be what finally gets sent to Honeycomb.

As an example, say we have some HTTP requests that provide sensitive values which have been added to a span. This code will examine all spans just before they are sent to Honeycomb and remove the sensitive values.

const beeline = require("honeycomb-beeline");
beeline({
  presendHook: function(ev) {
    // either:
    ev.data.scrubMe = undefined;
    // or:
    // delete ev.data.scrubMe
  }
});
beeline.finishTrace(beeline.startTrace({ name: "foo", scrubMe: "sensitive data" }));

Proxy Configuration 

If the environment variables HTTPS_PROXY or https_proxy are set, the Beeline will pick up and configure itself to use the proxy for all event traffic to Honeycomb.

Sampling Events 

We have built-in support for sampling in the Beeline. Simply set the sampleRate variable to your beeline configuration. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events:

require("honeycomb-beeline")({
  writeKey:    "YOUR_API_KEY",
  serviceName: "your-service-name",
  sampleRate: 5 // This will send 1/5 (20%) of events
  /* ... additional optional configuration ... */
});

The value of sampleRate must be a positive integer.

Sampling by default will apply the same sampling decision to all spans in a trace, so adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent. If sampling across multiple Beeline-instrumented services, set the same sample rate in all beeline configurations to avoid breaking traces.

Customizing Sampling Logic 

The samplerHook configuration option is available to customize the logic used for deterministic sampling. To avoid breaking traces with the custom samplerHook option, ensure that sampling logic is applied to data that all spans within a trace will have (such as trace.trace_id). If needed, you may promote span fields to trace-level (making them available on all spans within a trace) by calling addTraceContext() at the beginning of a trace, for instance: beeline.addTraceContext({ request_route: '/x/alive' }).

A custom samplerHook must return an object with this structure:

{
  shouldSample: boolean, // false will drop the event, true will keep it
  sampleRate: number // optional, will be reported to honeycomb
}

For example, assume you have instrumented an HTTP server. You would like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so:

const beeline = require("honeycomb-beeline");
const createHash = require("crypto").createHash;

beeline({
    writeKey:  "YOUR_API_KEY",
  serviceName: "your-service-name",
    samplerHook: samplerHook,
});

// deterministicSampler function based on https://github.com/honeycombio/beeline-nodejs/blob/main/lib/deterministic_sampler.js
function deterministicSampler(traceId, sampleRate) {
  const MAX_UINT32 = Math.pow(2, 32) - 1;
  const sum = createHash("sha1")
      .update(traceId)
      .digest();
  const upperBound = (MAX_UINT32 / sampleRate) >>> 0;
  return sum.readUInt32BE(0) <= upperBound;
}

function samplerHook(data) {
  // default sample rate to 10
  let sampleRate = 10;

  // default sampling decision to deterministic based on trace_id
  let shouldSample = deterministicSampler(data["trace.trace_id"], sampleRate);

  // drop all health checks requests
  if (data["request.path"] === "/x/alive") {
      shouldSample = false;
      sampleRate = 0;
  }
  return {
      shouldSample,
      sampleRate
  }
}

app.get('/x/alive', function(req, res) {
  res.send("I'm alive");
});

Note: Defining a sampling hook overrides the default deterministic sampling behavior for trace IDs. For head-based sampling behavior across a more complicated trace than the health check example above, you must ensure sampling decisions are based on a value all spans will have (such as trace_id or a custom trace context as we did above). Otherwise, you will get incomplete traces.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Automatic instrumentation supports trace propagation automatically, as long as your services are using the Honeycomb beeline, and an instrumented component to send and receive requests (express and https).

Interoperability With OpenTelemetry 

Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the W3C Trace Context specification.

To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers.

The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used.

To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services.

Note: Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed.

To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration.

An httpTraceParserHook is a function that takes an HTTP request as an argument and returns a Honeycomb trace context object. The HTTP request is provided to the function so that the author can make decisions about whether to trust the incoming headers based on information contained in the request (for example, perhaps you do not want to accept headers from the public internet).

An httpTracePropagationHook is a function that takes a Honeycomb trace context object as an argument and returns a map of name, value pairs representing serialized headers.

This example adds these hooks to parse and propagate W3C headers:

const beeline = require('honeycomb-beeline');

beeline({
    writeKey:    'YOUR_API_KEY',
    serviceName: 'your-service-name',
    httpTraceParserHook: beeline.w3c.httpTraceParserHook,
    httpTracePropagationHook: beeline.w3c.httpTracePropagationHook
});

Troubleshooting The Beeline 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

The Events I Am Generating Do Not Contain The Content I Expect 

Enable debug output (sent to STDOUT) by setting DEBUG=honeycomb-beeline:* in your environment. This will print the JSON representation of events to STDOUT as well as other debug level details and errors. This lets you quickly see what is getting sent and allows you to modify your code accordingly.

DEBUG=honeycomb-beeline:*

My Traces Are Showing Missing Root Spans 

There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause “missing span” gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service’s beeline configuration that only Honeycomb propagation headers should be parsed.

To override undesired W3C trace header propagation behavior, configure the Beeline to use an httpTraceParserHook:

const beeline = require("honeycomb-beeline");

beeline({
    writeKey:    "YOUR_API_KEY",
    serviceName: "your-service-name",
    httpTraceParserHook: beeline.honeycomb.httpTraceParserHook,
});

The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See Distributed Trace Propagation for more details.

Example Event 

Here is a sample event created by the Node.js Beeline:

{
  "Timestamp": "2018-03-20T00:47:25.339Z",
  "request.base_url": "",
  "request.fresh": false,
  "request.host": "localhost",
  "request.http_version": "HTTP/1.1",
  "request.remote_addr": "127.0.0.1",
  "request.method": "POST",
  "request.original_url": "/checkValid",
  "request.path": "/checkValid",
  "request.scheme": "http",
  "request.query": "{}",
  "request.secure": false,
  "request.url": "/checkValid",
  "request.xhr": true,
  "response.status_code": "200",
  "meta.instrumentation_count": 4,
  "meta.instrumentations": "[\"child_process\",\"express\",\"http\",\"https\"]",
  "meta.type": "express",
  "meta.version": "4.16.3",
  "meta.beeline_version": "1.0.2",
  "meta.node_version": "v9.10.0",
  "totals.mysql2.count": 2,
  "totals.mysql2.duration_ms": 13.291,
  "totals.mysql2.query.count": 2,
  "totals.mysql2.query.duration_ms": 13.291,
  "trace.trace_id": "11ad83a2-ca8d-4918-9db2-27524456d9f7",
  "trace.span_id": "4a3892ba-0936-46e1-8e17-31b887326027",
  "name": "request",
  "service_name": "express",
  "duration_ms": 15.229326
}

Queries To Try 

Here are some examples to get you started querying your app’s behavior:

Which Endpoints Are The Slowest? 

• Group By url
• Visualize the P99 of duration_ms values
• Where meta.type == express
• Order by P99(duration_ms) in descending (DESC) order

Where Is My App Spending The Most Time? 

• Group By meta.type
• Visualize the P99 of duration_ms values
• Order by P99(duration_ms) in descending (DESC) order

Which Users Are Using The Endpoint I Would Like To Deprecate? 

• Group By user.email
• Visualize the overall COUNT
• Where url == your endpoint url

Which XHR Endpoints Take The Longest? 

• Group By url
• Visualize the P99 of duration_ms values
• Where meta.type == express and xhr == true
• Order by P99(duration_ms) in descending (DESC) order

Contributions 

Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!

All contributions will be released under the Apache License 2.0.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit