Instrumentation | Honeycomb


The goal of Honeycomb instrumentation is to give you a language for sending wide, structured events into Honeycomb, so that they can be queried later.

What is an Event? 

Abstractly, an event is anything that happens in your system that is worth tracking. Some common choices of events include http requests to a server, a query to an SQL database, or a single step of a build process.

We say that events should be wide because they can contain many different fields. You can use the fields to associate the events with all of the additional data that it takes to understand what your system did with that event. There are traditional fields – client IP address, latency, server – but you can also add others. Did your system need to go through authorization? Did the result hit a cache? Did it go through a branch of an A/B test? Did it get a transient warning? What build of the server code was it using? Which AWS zone hosted it? Any data that can make sense of the event can be accumulated and attached to the event.

But at the end of the day, what you will send to Honeycomb is simply a JSON object that you POST to our API.

Events get ingested into our storage engine for later querying. Honeycomb’s fast storage engine serves queries in seconds, with no need to define schemas or indexes ahead of time.

The Honeycomb query engine allows you to ask all sorts of questions of your data. You can filter on any field, or combination of fields, to narrow down on a single page, or even on any user. You can group by error code or error message to spot common or recently occurring problems. You could SUM time spent on certain MySQL queries to identify if there are slow queries, but also if there are queries that are running too often.

Requests and Tracing 

One of the most common Honeycomb use cases is to send one event for each HTTP or RPC request in your system. You might even have several services (e.g., A, B, and C) that are touched in the lifecycle of serving what is one request to the end user, and they all send events to Honeycomb.

Example execution lifecycle

You can then do all sorts of queries within Honeycomb to answer any question under the sun. For instance, you could group by error code or error message to spot common or recently occurring problems fast. You could SUM time spent on certain MySQL queries to identify if there are not just slow queries, but queries that are running too often.

Not only that, but this model allows you to take advantage of distributed tracing.

If you set events to have the proper fields, you can visualize a full waterfall of how a given request flows through the system. That can make it straightforward to spot slow services and sections of code.

Example trace

Many of the Honeycomb automatic integrations, including OpenTelemetry add this data automatically, so traces are visible out of the box.

Batch Jobs and Serverless 

You can instrument background jobs to gain insight into how they operate. For example, you can send an event for every item that a processing job touches.

Honeycomb provides a buildevents integration for common Continuous Integration (CI) systems, including Travis-CI, CircleCI, and Jenkins. You can see which parts of your build are taking the most time, and streamline dependencies.

Execution flow of a batch job

Similarly, serverless requests can be instrumented; a Lambda invocation could easily send a Honeycomb event either using our bindings for that language. Learn more about Honeycomb’s AWS Lambda integrations.

Example serverless flow execution


Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications’ logs, cluster logs, and resource metrics. This data answers questions like:

  • How did response time change after a canary deployment?
  • How does application performance vary with container resource limits?
  • Are application errors happening on specific nodes, or across the fleet?

Learn more about Honeycomb’s Kubernetes integrations.

Ways to Get Started 


The automatic instrumentation in OpenTelemetry understand the standard packages you are using, then instrument them to send useful events to Honeycomb. There is no manual instrumentation required to generate basic events but with a little optional configuration, you can include your own fields, too. You can use this scaffolding to instrument your own business logic and attributes of your own service.

We have OpenTelemetry support for Go, Python, Node.js, Ruby, Java, .NET, and others.

CURL, SDKs, and Honeytail 

Need to stay closer to the metal? We do not blame you! You can POST data directly to us with our Events API, and we offer a structured logging library in a variety of languages. Or, you can send us structured text logfiles with Honeytail.

Example Applications 

Our Examples GitHub repository links to a wide range of instrumented sample applications that illustrate how to generate custom events and send them to Honeycomb.