Connecting Fluentd to Honeycomb | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Connecting Fluentd to Honeycomb

Fluentd is a widely-used data router. If you’re using Fluentd to aggregate structured logs, Fluentd’s out_http plugin makes it easy to forward data to Honeycomb.

Getting Started  🔗

To set up the plugin, first grab your team API key from your Honeycomb account page, and then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf). A basic configuration to forward events with the my.logs tag to Honeycomb looks like this:

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/fluentd_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Set Event Timestamps  🔗

In Fluentd, each event has a distinguished time attribute. In general, you will use a parser plugin to extract the time attribute from log lines. You can read more about the structure of a Fluentd event here.

For example, if you have a JSON log file containing timestamps in the format:

{"timestamp": "2018-02-04T14:55:10Z", "host": "app22", ...}

Then, you would extract the time value using the following Fluentd configuration:

<source>
  @type tail
  path /var/log/my.logs
  <parse>
    @json # Use the JSON parser plugin to parse records
    time_key timestamp # Extract the time value from the `timestamp` key
    time_type string # Expect a string timestamp
    time_format %Y-%m-%dT%H:%M:%SZ # Specify the timestamp format
  </parse>
  tag my.logs
</source>

<filter my.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    data ${ record }
    time ${ time.iso8601() }
  </record>
</filter>

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/myapp_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>