Connecting Fluentd to Honeycomb | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Connecting Fluentd to Honeycomb

Fluentd is a widely-used data router. If you’re using Fluentd to aggregate structured logs, Fluentd’s out_http plugin makes it easy to forward data to Honeycomb.

Getting Started  🔗

To set up the plugin, first grab your team API key from your Honeycomb account page, and then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf). A basic configuration to forward events with the my.logs tag to Honeycomb looks like this:

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/fluentd_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

More Advanced Configuration  🔗

Fluentd supports a wide variety of logging setups. Here are some things you might want to do:

Manage Multiple Data Sources  🔗

If you have completely unrelated log sources, you can forward them to separate datasets by using two separate configuration blocks:

<match a.*>
  @type http
  endpoint https://api.honeycomb.io/1/batch/dataset_a
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

<match b.*>
  @type http
  endpoint https://api.honeycomb.io/1/batch/dataset_b
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Set Event Timestamps  🔗

In Fluentd, each event has a distinguished time attribute. In general, you will use a parser plugin to extract the time attribute from log lines. You can read more about the structure of a Fluentd event here.

For example, if you have a JSON log file containing timestamps in the format:

{"timestamp": "2018-02-04T14:55:10Z", "host": "app22", ...}

Then, you would extract the time value using the following Fluentd configuration:

<source>
  @type tail
  path /var/log/myapp.log
  <parse>
    @json # Use the JSON parser plugin to parse records
    time_key timestamp # Extract the time value from the `timestamp` key
    time_type string # Expect a string timestamp
    time_format %Y-%m-%dT%H:%M:%SZ # Specify the timestamp format
    keep_time_key true # Keep the time key field in the record
  </parse>
</source>

<match myapp.*>
  @type http
  endpoint https://api.honeycomb.io/1/batch/myapp_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

The fluentd json formatter does not serialize the tag and time fields. Setting the keep_time_key to true is recommended to ensure the timestamp field remains in the record that is sent to Honeycomb. With this configuration, each event sent to Honeycomb will have the value of the timestamp field as its timestamp, rather than the time at which the event was parsed.