We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Connecting Heka to Honeycomb

Mozilla’s Heka is intended to collect data from many sources and provide a unified interface for dealing with it. If you have an existing Heka pipeline, sending a stream of data in to Honeycomb is amazingly easy!

Data expectations  🔗

Honeycomb needs messages to arrive in a flat JSON format. If you already have log messages entering Heka in JSON format, great! You can pipe them right through. If you don’t, Heka’s powerful set of encoders and decoders are there for you. There are decoders that will automatically understand common log formats (nginx, mysql, rsyslog, gelf, etc.) and turn them into a Heka message that you can then re-pack in to JSON in the encoding section.

Manipulating payloads  🔗

Heka’s sandboxed plugins like the JSON Payload Transform plugin make it easy to massage the messages as they traverse Heka. If you want to modify the payload to add (eg hostname or environment) or remove (eg sensitive) fields before submitting the message to Honeycomb, this is an easy place to do it. And if you want to do more complicated things, you can always write a small bit of Lua to get it done!

Transmitting events to Honeycomb  🔗

Finally, we get to the HTTP output module, which contains everything necessary to wrap up the message and send it in to the Honeycomb API. You’ll need to configure it to:

It’s likely that you’ll have more messages flowing through Heka than you’re interested in sending to Honeycomb, so adding a Message Matcher to the HTTP output module will select only those messages that you want to see in Honeycomb for transmission.

Here’s a sample section of a Heka TOML config file to send traffic that matches application logs (field app_source is application) and sends them on to the application dataset in Honeycomb.

type = "HttpOutput"
message_matcher = "Fields[app_source] == 'application'"
encoder = "PayloadEncoder"
address = "https://api.honeycomb.io/1/events/application"

X-Honeycomb-Team = ["YOUR_API_KEY"]