Fluentd is a widely-used data router.
If you are using Fluentd to aggregate structured logs, Fluentd’s out_http plugin makes it easy to forward data to Honeycomb.
To set up the plugin, first grab your team API key from your Honeycomb account page, and then update your Fluentd configuration file (usually found in /etc/fluentd/fluentd.conf or /etc/td-agent/td-agent.conf).
A basic configuration to forward events with the my.logs tag to the Honeycomb dataset fluentd_dataset looks like this:
<filter my.logs>
@type record_transformer
enable_ruby true
renew_record true
<record>
data ${ record }
time ${ time.iso8601() }
</record>
</filter>
<match my.logs>
@type http
endpoint https://api.honeycomb.io/1/batch/fluentd_dataset
headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
<format>
@type json
</format>
json_array true
<buffer>
flush_interval 2s
</buffer>
</match>
In Fluentd, each event has a distinguished time attribute.
In general, you will use a parser plugin to extract the time attribute from log lines.
You can read more about the structure of a Fluentd event here.
For example, if you have a JSON log file containing timestamps in the format:
{"timestamp": "2018-02-04T14:55:10Z", "host": "app22", ...}
Then, you would extract the time value using the following Fluentd configuration:
<source>
@type tail
path /var/log/my.logs
<parse>
@json # Use the JSON parser plugin to parse records
time_key timestamp # Extract the time value from the `timestamp` key
time_type string # Expect a string timestamp
time_format %Y-%m-%dT%H:%M:%SZ # Specify the timestamp format
</parse>
tag my.logs
</source>
<filter my.logs>
@type record_transformer
enable_ruby true
renew_record true
<record>
data ${ record }
time ${ time.iso8601() }
</record>
</filter>
<match my.logs>
@type http
endpoint https://api.honeycomb.io/1/batch/myapp_dataset
headers {"X-Honeycomb-Team":"YOUR_API_KEY"}
<format>
@type json
</format>
json_array true
<buffer>
flush_interval 2s
</buffer>
</match>
Did you find what you were looking for?