Connecting Fluentd to Honeycomb

Fluentd is a widely-used data router. If you are using Fluentd to aggregate structured logs, Fluentd’s out_http plugin makes it easy to forward data to Honeycomb.

Getting Started

To set up the plugin, first grab your team API key from your Honeycomb account page, and then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf). A basic configuration to forward events with the my.logs tag to the Honeycomb dataset fluentd_dataset looks like this:

<filter my.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    data ${ record }
    time ${ time.iso8601() }
  </record>
</filter>

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/fluentd_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Set Event Timestamps

In Fluentd, each event has a distinguished time attribute. In general, you will use a parser plugin to extract the time attribute from log lines. You can read more about the structure of a Fluentd event here.

For example, if you have a JSON log file containing timestamps in the format:

{"timestamp": "2018-02-04T14:55:10Z", "host": "app22", ...}

Then, you would extract the time value using the following Fluentd configuration:

<source>
  @type tail
  path /var/log/my.logs
  <parse>
    @json # Use the JSON parser plugin to parse records
    time_key timestamp # Extract the time value from the `timestamp` key
    time_type string # Expect a string timestamp
    time_format %Y-%m-%dT%H:%M:%SZ # Specify the timestamp format
  </parse>
  tag my.logs
</source>

<filter my.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    data ${ record }
    time ${ time.iso8601() }
  </record>
</filter>

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/myapp_dataset
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Honeycomb.io Documentation

Connecting Fluentd to Honeycomb

Getting Started

Set Event Timestamps