Skip to main content
Libhoney for Python is Honeycomb’s structured logging library for Python applications. It is a low-level library that helps you send structured events to Honeycomb’s Events API.
If you are instrumenting a new application for tracing, we recommend that you use OpenTelemetry instead.

​
Installation

  pip install libhoney

​
Initialization

Initialize the library by passing in your Team API key and the default dataset name to which it should send events. 
import libhoney

libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True)

# ... Do work and capture events

​
Working With Proxies

Using a proxy requires overriding the default Transmission implementation when initializing libhoney: 
import libhoney

libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True,
              transmission_impl=libhoney.transmission.Transmission(proxies={'https': 'https://myproxy.example.com'}))
The proxies map passed in is documented in the requests documentation. Note that if you override transmission_impl, if you also have non-default values for options such as max_concurrent_batches and max_batch_size in libhoney.init, they will need to be specified in the new Transmission object. Further configuration options can be found in the init.py file.

​
Building And Sending Events

Once initialized, libhoney is ready to send events. Events go through three phases: 
# Creation

event = libhoney.new_event()

# Adding Fields
event.add_field("key", "val")
dataMap = {"key2": "val2"}
event.add(dataMap)

# Enqueuing
event.send()
Upon calling .send(), the event is dispatched to be sent to Honeycomb. Events are queued to be transmitted asynchronously, allowing your application to continue without delay. The queue is limited in size, however, and when creating events faster than they can be sent, overflowed events will be dropped instead of backing up and slowing down your application. This behavior can be configured at initialization by adjusting block_on_send, as described here. In its simplest form, you can add a single field to an event with the libhoney.add_field(k, v) method. If you add the same key multiple times, only the last value added will be kept. More complex structures (dicts and objects—things that can be serialized into a JSON object) can be added to an event with the .add(data) method. Events can have metadata associated with them that is not sent to Honeycomb. This metadata is used to identify the event when processing the response. More detail about metadata is below in the Response section.

​
Handling Responses

Sending an event is an asynchronous action and will avoid blocking by default. .send() will enqueue the event to be batched and sent as soon as possible (thus, the return value doesn’t indicate that the event was successfully sent). You can monitor the queue returned by .responses() to check whether events were successfully received by Honeycomb’s servers. The responses queue will receive responses for each batch of events sent to the Honeycomb API. Before sending an event, you have the option to attach metadata to that event. This metadata is not sent to Honeycomb; instead, it is used to help you match up individual responses with sent events. When sending an event, libhoney will take the metadata from the event and attach it to the response object for you to consume. Add metadata by calling .add_metadata({"key": "value"}) on an event. Responses are represented as dicts with the following keys:
  • metadata: the metadata you attached to the event to which this response corresponds
  • status_code: the HTTP status code returned by Honeycomb when trying to send the event. 2xx indicates success.
  • duration: the number of milliseconds it took to send the event.
  • body: the body of the HTTP response from Honeycomb. On failures, this body contains some more information about the failure.
  • error: when the event does not even get to create a HTTP attempt, the reason will be in this field. For example, when sampled or dropped because of a queue overflow.
You do not have to process responses if you are not interested in them—simply ignoring them is perfectly safe. Unread responses will be dropped.

​
What to Send?

Honeycomb events are composed of fields of four basic types: string, int, float, and bool. You should send any data that will help you provide context to an event in your application, such as user ids, timers, measurements, system info, build ID, and metadata. For more guidance about instrumenting your application, check our instrumentation introduction guide.

​
Examples

​
Simple: Send a Blob of Data

import libhoney

libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True)

# create a new event
ev = libhoney.new_event()
# add data up front
ev.add({
  "method": "get",
  "hostname": "appserver15",
  "payload_length": 27
})

# do some work, maybe take some measurements

# add another field
# you can send raw measurements - Honeycomb can help us compute averages, percentiles, etc
# when you query
ev.add_field("duration_ms", 153.12)
ev.send()

​
Intermediate: Override Some Attributes

# ... Initialization code ...
params = {
  "hostname": "foo.local",
  "built": false,
  "user_id": -1
}

# add the above parameters to the global client - these will get added
# to every new event, but can be overridden
libhoney.add(params)

# Spawn a new event and override the user_id
event = libhoney.new_event()
# overrides a previously set field
event.add_field("user_id", 15)
# new fields are also welcome
event.add_field("latency_ms", datetime.datetime.now() - start)
event.send()
Further examples can be found on GitHub.

​
Middleware Examples: Django

Django is a widely-used, high-level Python Web framework. Each inbound HTTP request as received by a framework like Django maps nicely to Honeycomb events, representing “a single thing of interest that happened” in a given system. Django middlewares are simply classes that have access to the request object (and optionally the response object) in the application’s request-response chain. As such, you can define a simple honey_middleware.py file as in the following: 
import os
import time
import libhoney

class HoneyMiddleware(object):
  def __init__(self):
    libhoney.init(writekey=os.environ["YOUR_API_KEY"],
                  dataset="django-requests", debug=True)


  def process_request(self, request):
    request.start_time = time.time()

    return None


  def process_response(self, request, response):
    response_time = time.time() - request.start_time

    ev = libhoney.Event(data={
      "method": request.method,
      "scheme": request.scheme,
      "path": request.path,
      "query": request.GET,
      "isSecure": request.is_secure(),
      "isAjax": request.is_ajax(),
      "isUserAuthenticated": request.user.is_authenticated(),
      "username": request.user.username,
      "host": request.get_host(),
      "ip": request.META['REMOTE_ADDR'],
      "responseTime_ms": response_time * 1000,
    })
    ev.send()

    return response
See the examples/ directory on GitHub for more sample code demonstrating how to use events, builders, fields, and dynamic fields, specifically in the context of Django middleware.

​
Advanced Usage: Global Fields

Some fields are interesting to have on every event: the build ID of the application, the application server’s hostname, or any fields you would like. Rather than remembering to create these for each event you create, you can add them globally at any time in your application. 
libhoney.add_field("build_id", 12345)

# can also add a dictionary of fields
libhoney.add({
  "server_hostname": "myHostname",
  "server_ip": "1.2.3.4",
})

# now any event I create has these fields
ev = libhoney.new_event()
ev.send()

​
Advanced Usage: Utilizing Builders

Global fields can be useful, but what if you want to propagate common fields that are only relevant to one component of your application? Builders are objects that generate new events, but that also propagate any fields added to it to the created events. You can also clone builders—the cloned builder will have a copy of all the fields and dynamic fields in the original. As your application forks down into more and more specific functionality, you can create more detailed builders. The final event creation in the leaves of your application’s tree will have all the data you have added along the way in addition to the specifics of this event. The global scope is essentially a specialized builder, for capturing attributes that are likely useful to all events (for example, hostname or environment). Adding this kind of peripheral and normally unavailable information to every event gives you enormous power to identify patterns that would otherwise be invisible in the context of a single request. 

libhoney.add_field("name", "fruit stand")

# this part of the application deals with apple sales
b1 = libhoney.NewBuilder()
b1.add_field("fruit", "apple")

ev1 = b1.new_event()
# will have field { "name": "fruit stand", "fruit": "apple" }
ev1.send()

# ...

# this component is specific to medium fuji apple sales - clone the previous builder
# and add more context to events from this component
b2 = b1.clone()
b2.add_field("variety", "fuji")
b2.add_field("size", "medium")

# ...

ev2 = b2.new_event()
ev2.add_field("qty_sold", 5)
# will have field { "name": "fruit stand", "fruit": "apple", "variety": "fuji", "size": "medium", "qty_sold": 5 }
ev2.send()

​
Advanced Usage: Dynamic Fields

The top-level libhoney and Builders support .add_dynamic_field(func). Adding a dynamic field to a Builder or top-level libhoney ensures that each time an event is created, the provided function is executed and the returned value is added to the event. The key is the __name__ attribute of the function. This may be useful for including dynamic process information such as memory used, number of threads, concurrent requests, and so on to each event. Adding this kind of dynamic data to an event makes it easy to understand the application’s context when looking at an individual event or error condition. 
import random

colors = ['red', 'blue', 'green', 'yellow']

def color_category():
  return colors[random.randint(0,3)]

libhoney.add_dynamic_field(color_category)

ev = libhoney.new_event()
# will contain field 'color_category' with one value in 'red|blue|green|yellow'
ev.send()

​
Advanced Usage: Responses Queue

If you would like to view the API response for events you have sent, if you need to verify receipt of an event, you can use the responses queue to do so: 

ev = libhoney.new_event()
# metadata that is attached to an event is preserved in the responses queue
ev.add_metadata({"my": "metadata", "event_id": 12345})
ev.send()

def read_responses(resp_queue):
    '''read responses from the libhoney queue, print them out.'''
    while True:
        resp = resp_queue.get()
        # libhoney will enqueue a None value after we call libhoney.close()
        if resp is None:
            break
        status = "sending event with metadata {} took {}ms and got response code {} with message \"{}\" and error message \"{}\"".format(
            resp["metadata"], resp["duration"], resp["status_code"],
            resp["body"].rstrip(), resp["error"])
        print(status)

# ...

read_responses(libhoney.responses())
Note that responses, if unread, will begin to fill the queue. Once full, new responses will be dropped. To change this behavior, set block_on_response=True when initializing the SDK with libhoney.init() - this will prevent dropped responses, but will also prevent new batches from being sent if the queue is not being read from. To continuously process the responses queue, you can run your processing function in another thread. 
t = threading.Thread(target=read_responses, args=(libhoney.responses(),))
t.start()

​
Customizing Event Transmission

By default, events are sent to the Honeycomb API. It is possible to override the default transmission implementation by specifying transmission_impl to init. A couple of alternative implementations ship with the SDK.
  • FileTransmission: writes events to a file handle, defaulting to stderr
  • TornadoTransmission: sends events using Tornado’s AsyncHTTPClient rather than using a thread pool
To override the default transmission and write events out to stderr: 

import libhoney
from libhoney.transmission import FileTransmission

libhoney.init(writekey='YOUR_API_KEY', transmission_impl=FileTransmission(output=sys.stderr))
If you have unique requirements, you can write your own implementation. See the transmission source to get started.

​
Flushing Events

The Python SDK has no mechanism for sending an individual event immediately - events are enqueued and sent asynchronously in batches. You can, however, trigger a flush of the send queue. There are two ways to do this. 
libhoney.flush()
.flush() instructs the transmission to send all of its events and will block until all events in the queue have been sent (or an attempt has been made - see the responses queue if you need to verify success). 
libhoney.close()
.close() flushes all events, but prevents transmission of new events. This is best called as part of your application’s shutdown logic to ensure that all events are sent before the program terminates.

​
Troubleshooting

Refer to Common Issues with Sending Data in Honeycomb.

​
Contributions

Features, bug fixes and other changes to libhoney are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file! All contributions will be released under the Apache License 2.0.