SDK spec | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

SDK spec

This document describes the requirements to which all Honeycomb client libraries must comply. The implementation and syntax for each of these requirements will differ by language.

A Honeycomb Client Library is intended to be used from within the code of an application. Its primary purpose is to make it easy to create events, add data to events, and send those events to Honeycomb. It must impact the performance of the application as little as possible, especially in failure scenarios. It may include extra functionality that makes it easier to add specific types of data to an event (for example, timers), but the focus should be on providing core functionality and allowing people to wrap the library for their convenience functions.

Throughout this guide, the following vocabulary is used:

  • must: if the library does not do this, it will not be accepted
  • should: it is suggested the library behave this way, but, with a good explanation why, may be accepted without.
  • may: adding this functionality is nice but not required

To be accepted as a client library, all criteria in the Minimum Necessary section must be met. The library must support:

  • initialization
  • creating events, adding data, and triggering event send
  • asynchronous transmission of those events
  • automated testing core functionality
  • released under the Apache License 2.0

The Fully Featured Library section details the complete spec for the most complete libraries out there. Only a few languages have libraries that meet everything outlined in the fully featured library spec.

Minimum Necessary  🔗

Initialization  🔗

The library must have an initialization function to call. It should be an error to send events on an uninitialized library.

The initialization should start up any background threads necessary for transmission. The number of threads to use, the size of the pending and response queues, blocking, and so on should be able to be configured with the initialization.

You should be able to set any of the following during initialization:

  • Team Writekey (no default)
  • Dataset (no default)
  • API Host (default to https://api.honeycomb.io)
  • Sample Rate (defaults to 1)

We intentionally allow initialization without a writekey or dataset but will fail on event send as a result.

Events  🔗

You must be able to create an event, add multiple fields to it, and send the event.

  • Creation: you should create an event by instantiating a class or calling a constructor such as new_event()

  • Adding fields: you must be able to add a single key/value pair with a function named something like add_field(k,v)

  • Sending: you must be able to send an individual event by calling a send() method on the event

  • Honeycomb serializes nested JSON structures into strings at the API layer. Since you can change that setting in the web UI as noted below, do not use the SDK to transform the event payload (or serialize values) before sending. By default:

    • Arrays and nested objects will be serialized into strings.
    • Numbers will be interpreted as floats.
    • Booleans will be identified as such.

    You can configure Honeycomb to unpack nested JSON objects. See “Data Expectations” for sending JSON for more details.

You must be able to set the following attributes on an event:

  • Team Writekey
  • Dataset
  • APIHost
  • Sample Rate
  • Timestamp (default: current time)

If not set on the event, each of these attributes should default to what was set during library initialization.

Transmission  🔗

Sending events must be non-blocking by default. When triggering an event send, the event should be handed to a transmission component and the send() method should return immediately.

Sampling happens in the send() function - if the Sample Rate is an integer greater than 1, send() should only send events with a probability of 1/Sample Rate.

Transmission must use the parameters from the event being sent for the actual sending - it should not use or check any globally set attributes.

Transmission must send an event to the APIHost specified on the event using HTTP POST. The key/value pairs that comprised the data for the event must be JSON-encoded as the body of the POST.

The POST must be made to /1/events/<dataset_name> or, if using the batch endpoint, /1/batch/<dataset_name>

The POST must include the following HTTP headers:

  • X-Honeycomb-Team: the event’s writekey
  • Content-Type: application/json
  • User-Agent: a string describing the library. Should take the form libhoney-<lang>/<version>. For example, libhoney-py/1.0.0 for version 1.0.0 of the python library.

The POST should include the following HTTP headers if using the /1/events endpoint:

  • X-Honeycomb-Event-Time: the time the event took place. If unset, defaults to the time the server receives the event. Time must be sent using RFC3339 format, which uses the character T to separate the date and time
  • X-Honeycomb-Samplerate: the sample rate for the event. If unset, defaults to 1

The POST must use the appropriate JSON format as described in the Honeycomb REST API documentation

Tests  🔗

Tests should cover:

  • initializing the library
  • creating, updating, and sending events
  • events inheriting properties from the global state
  • events inheriting properties from the builder, if implemented
  • transmission using attributes on the events
  • all events generate a response, even in error conditions (if responses are implemented)

Documentation  🔗

The library must provide documentation in three ways:

  1. code comments must exist and describe each public function. Expect that these comments will be used to generate API documentation in a method similar to godoc or pydoc
  2. a README that describes the library at a higher level.
  3. an example file that successfully runs using the library to submit a few events to Honeycomb

License  🔗

The code must be released under the Apache License 2.0 as described here: http://choosealicense.com/licenses/apache-2.0/

Initialization  🔗

The library must have an initialization function to call. It should be an error to send events on an uninitialized library.

The initialization should start up any background threads necessary for transmission. The number of threads to use, the size of the pending and response queues, blocking, and so on should be able to be configured with the initialization.

You must be able to set the following during initialization:

You may be able to initialize a null transmission, in which case no events will be sent (for example, for testing or non-production environments).

Events  🔗

The minimum data required to send a valid event must be a (possibly inherited) single key/value pair. It is an error to attempt to send an event with no data, no APIHost, a missing writekey, or a missing dataset name.

The basic flow for an event is:

  • creation: either create an instance of a class or provide a function to return a new event. For example, new_event()
  • adding data one or more times
    • must have a way of adding a single key/value pair. This method should be named something like add_field(k,v)
      • must be able to call add_field(k,v) multiple times on the same event with different keys in order to add multiple key/value pairs to the same event
    • may have a way of adding a more complex object (for example, dict, struct, or map) that should be named add(obj). This method must break down objects and add their components (For example, add each key/value pair in a python dict as distinct fields).
  • send: The event should be dispatched to Honeycomb via a send() or send_presampled() method.

Internally, an event must include at least the following attributes:

  • A container for all the key/value pairs that make up the event. This container is not intended to be accessible by the user’s code. This is the component that will be JSON-encoded and sent to Honeycomb. The library must not add any content to this container other than the key/value pairs and dynamic values added by the user (from any scope).

  • Honeycomb serializes nested JSON structures into strings at the API layer. Since you can change that setting in the web UI as noted below, do not use the SDK to transform the event payload (or serialize values) before sending. By default:

    • Arrays and nested objects will be serialized into strings.
    • Numbers will be interpreted as floats.
    • Booleans will be identified as such.

    You can configure Honeycomb to unpack nested JSON objects. See “Data Expectations” for sending JSON for more details.

Internally, an event must include the following attributes, and allow each to be overridden on a per-event basis. All of these values should be initialized to those set in the global state or builder, and must be set before handing the event off to Transmission:

  • The sample rate for this event.
  • The writekey for the Honeycomb team
  • The dataset to which this event should be sent
  • the APIHost to use to POST

Internally, an event must include the following attributes, and allow each to be overridden on a per-event basis. These should not inherit values from the builder or global state.

  • The time this event happened (might be in the past). Should default to now if it is not explicitly set.
  • metadata, to be handed back to the user via a response. This data is not sent to Honeycomb. If not set, it should remain empty.

If you add the same key to an event multiple times with different values, only the finally added value is kept and sent with the event.

Sending events must be non-blocking by default. More on this in the Transmission section below. Modifying an individual event does not need to be threadsafe.

Builders  🔗

Builders are optional.

A builder is an event factory - builders are used to create events with specific properties. A builder contains key/value pairs or dynamic fields and every event generated from that builder inherits those key/values as initial state. A builder must also allow writekey, dataset, sample rate, and APIHost to be set on the builder and events created from the builder will inherit those settings. The settings in a builder override those in the global state for events created from that builder.

Builders may be created by either instantiating a class or using a constructor. The constructor should be named something like new_builder()

The semantics of creating builders and adding data to builders should be the same as creating and adding data to events. A builder should have an additional method to create an event using the state of the builder, named something like new_event()

A dynamic field is a function which, when called, returns a key/value pair. The user of the library provides the function, and it is called once for each event that is created from the builder.

Global state  🔗

You should be able to add key/value pairs to the global state. You may be able to add dynamic fields to the global state.

If there are any fields set in the global state, every event should inherit those fields.

Responses  🔗

The library should have a method of providing the user with a record of Honeycomb’s response to each event sent. Getting data on the responses should be asynchronous in nature - the response should not be tied to sending the event.

Reading responses must not be a requirement to efficiently send data to Honeycomb. The user may be able to choose to block sending on making sure the response queue is consumed.

Any content the user provides in the metadata should be handed back to the user along with the response. This metadata is the method by which the user can tie a specific event to its response.

Every event must have a corresponding Response, even those that are sampled or dropped due to overflows.

A response object must have the following attributes:

  • metadata: the data handed to the event to which this is the response.
  • duration: the amount of time it took to perform the HTTP POST to submit the event to Honeycomb. This duration does not include any time the event spent in the queue waiting to be sent.
  • status_code: the HTTP status code Honeycomb gave upon receiving the event. It should be only the numerical component (For example, 200, not 200 OK).
  • body: the body of the HTTP response from Honeycomb.
  • err: a string indicating the reason for any error or unsent event. err must be populated for events dropped due to sampling or queue overflow. It is not necessary to populate err based on the HTTP response from Honeycomb; those failed sends will have the reason in the body attribute.

A response may have the following attributes:

  • queue_duration: the amount of time spent in the queue waiting to be sent
  • total_duration: total time for the event, including time both spent in the queue and being sent. It does not include any time between event creation and when the event is enqueued to be sent. It does not include any time the response sits in the response queue

Transmission  🔗

Sending events must be non-blocking by default. When triggering an event send, the event should be handed to a transmission component and the send() method should return immediately.

Sampling happens in the send() function - if the Sample Rate is an integer greater than 1, send() should only send events with a probability of 1/Sample Rate.

Transmission should enqueue the event for eventual sending. It should have a configurable number of threads sending individual events. The default queue size should be 1,000 or 10,000. When the queue overflows, excess events should be dropped.

Transmission should use connection pooling.

Transmission should do batching internally, and if so, must have a method for triggering sending a batch based on either the entire batch filling or a timer triggering. The size of the batch and frequency of the timer must both be configurable. Sending may use the batching API endpoint for more efficient transmission. Suggested defaults are a batch size of 50 and a batch timer trigger of 100ms. When batching, the library must separate events into batches that all have API Host, writekey, and dataset in common, and send off batches as they fill.

Transmission may provide an option for eventually blocking sends. It should buffer a number of events before blocking. The size of the send queue must be configurable, and should default to 1,000 or 10,000. When blocking, an event send() should not return until the event is enqueued.

Transmission must use the parameters from the event being sent for the actual sending - it should not use or check any globally set attributes.

Transmission must send an event to the APIHost specified on the event using HTTP POST. The key/value pairs that comprised the data for the event must be JSON-encoded as the body of the POST.

The POST must be made to /1/events/<dataset_name> or, if using the batch endpoint, /1/batch/<dataset_name>

The POST must include the following HTTP headers:

  • X-Honeycomb-Team: the event’s writekey
  • Content-Type: application/json
  • User-Agent: a string describing the library. Should take the form libhoneycomb-<lang>/<version>. For example, libhoneycomb-py/1.0.0 for version 1.0.0 of the python library.

The POST should include the following HTTP headers if using the /1/events endpoint:

  • X-Honeycomb-Event-Time: the time the event took place. If unset, defaults to the time the server receives the event. Time must be sent using RFC3339 format, which uses the character T to separate the date and time
  • X-Honeycomb-Samplerate: the sample rate for the event. If unset, defaults to 1

The POST must use the appropriate JSON format as described in the Honeycomb REST API documentation

Sampling in the SDK  🔗

Sampling refers to selectively dropping events. The Sample Rate is representative of the ratio of events to drop. A sample rate of 5 means that 1 out of every 5 events will be sent. A sample rate of 1 means every event will be sent. Sample rates less than 1 are invalid and should be set to 1

Sampling must take place when you call an event’s send() method.

Sampling must be probabilistic. In other words, if the sample rate is 2, it does not mean that every other event must be sent. It means that each event should have a probability of 50% that it gets sent.

If an event that is subject to sampling (i.e. a sample rate greater than 1), any event that is not dropped must have the sample rate attribute for that event set.

The send_presampled() method should set the sample rate according to the event’s attributes, but should not do any sampling. send_presampled() is used when the calling function samples traffic so the library should pass all events through unchanged.

The page on sampling best practices explains how the Honeycomb engine uses sampled data.

Tests  🔗

Tests should cover

  • initializing the library
  • creating, updating, and sending events
  • events inheriting properties from the global state
  • events inheriting properties from the builder, if implemented
  • transmission using attributes on the events
  • all events generate a response, even in error conditions (if responses are implemented)

##3 Documentation

The library must provide documentation in three ways:

  1. code comments must exist and describe each public function. Expect that these comments will be used to generate API documentation in a method similar to godoc or pydoc
  2. a README that describes the library at a higher level.
  3. an example file that successfully runs using the library to submit a few events to Honeycomb

License  🔗

The code must be released under the Apache License 2.0 as described here: http://choosealicense.com/licenses/apache-2.0/