Forward Fluentd Data

Fluentd is a widely-used data router. If you are using Fluentd to aggregate structured logs, Fluentd’s out_http plugin makes it easy to forward data to Honeycomb.

If your system uses logspout as a log router for Docker containers, you can send logs to Honeycomb with one of the logspout third-party modules that integrates with logstash or fluentd.

Getting Started

To set up the plugin, first grab your team API key from your Honeycomb account page, and then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf). A basic configuration to forward events with the my.logs tag to the Honeycomb dataset fluentd_dataset looks like this:

<filter my.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    data ${ record }
    time ${ time.iso8601() }
  </record>
</filter>

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/fluentd_dataset <!--US instance-->
  <!--endpoint https://api.eu1.honeycomb.io/1/batch/fluentd_dataset--><!--EU instance-->
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Set Event Timestamps

In Fluentd, each event has a distinguished time attribute. In general, you will use a parser plugin to extract the time attribute from log lines. You can read more about the structure of a Fluentd event here. For example, if you have a JSON log file containing timestamps in the format:

{"timestamp": "2018-02-04T14:55:10Z", "host": "app22", ...}

Then, you would extract the time value using the following Fluentd configuration:

<source>
  @type tail
  path /var/log/my.logs
  <parse>
    @json # Use the JSON parser plugin to parse records
    time_key timestamp # Extract the time value from the `timestamp` key
    time_type string # Expect a string timestamp
    time_format %Y-%m-%dT%H:%M:%SZ # Specify the timestamp format
  </parse>
  tag my.logs
</source>

<filter my.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    data ${ record }
    time ${ time.iso8601() }
  </record>
</filter>

<match my.logs>
  @type http
  endpoint https://api.honeycomb.io/1/batch/myapp_dataset <!--US instance-->
  <!--endpoint https://api.eu1.honeycomb.io/1/batch/myapp_dataset--><!--EU instance-->
  headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Instrument Applications

Instrument Infrastructure

CI/CD Observability

Manage Pipelines

Shape Data

Control Data Volume

Use Cases

Getting Started

Set Event Timestamps

​Getting Started

​Set Event Timestamps

Getting Started

Set Event Timestamps