sudo initctl start honeytail
sudo systemctl start honeytail
honeytail -c /etc/honeytail/honeytail.conf
JSON is one of the most flexible formats in the data landscape we have today, and our JSON connector is perfect for your application’s custom log data.
Unstructured text logs are so 2009; whether you are primarily using Honeycomb, json over Logstash, or some other JSON-friendly service, pointing your existing logs at Honeycomb is simple.
Honeycomb expects data with a flat structure. By default, any structure deeper than top level keys will be serialized and a string representation of the content will be used in the field. However, Honeycomb can automatically unpack nested JSON objects and flatten them into unique columns. This is a per-dataset setting, and it is off by default. You must be a team owner to change this setting.
If you enable this setting, nested objects will be flattened with new fields and field names created based on the keys.
For example, {"outer": {"inner": 42}} would become a field outer.inner with a value of 42.
To tell Honeycomb to automatically unpack JSON objects:
Changing this setting takes effect within 60 seconds.
Note: If your objects are deeply-nested, unpacking may result in a very large number of columns in Honeycomb. Consider unpacking only to the level of columns you will find useful. Any objects nested more deeply than the depth you select here will be converted to strings under the last unpacked column. In particular, if nested structures in your data can be created/added by your users (for example, HTTP headers), consider not unpacking them to that level.
Download and install the latest honeytail by running:
Download the honeytail_1.8.2_amd64.deb package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail_1.8.2_amd64.deb
Verify the package.
echo 'c9101060b782c34045f359bfe7b7a0a95c6a54eb97ed51777b053a769afa823b honeytail_1.8.2_amd64.deb' | sha256sum -c
Install the package.
sudo dpkg -i honeytail_1.8.2_amd64.deb
The packages install honeytail, its config file /etc/honeytail/honeytail.conf,
and some start scripts.
Build honeytail from source if you need it in an unpackaged form or for ad-hoc use.
Download the honeytail_1.8.2_arm64.deb package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail_1.8.2_arm64.deb
Verify the package.
echo 'c5a03a2ef4b850be35fb3809173efd3d6e8b16c52b770f61691355b7d39505b0 honeytail_1.8.2_arm64.deb' | sha256sum -c
Install the package.
sudo dpkg -i honeytail_1.8.2_arm64.deb
The packages install honeytail, its config file /etc/honeytail/honeytail.conf,
and some start scripts.
Build honeytail from source if you need it in an unpackaged form or for ad-hoc use.
Download the honeytail-1.8.2-1.x86_64.rpm package.
wget -q https://honeycomb.io/download/honeytail/v1.8.2/honeytail-1.8.2-1.x86_64.rpm
Verify the package.
echo 'bccdedc9aa3776b827899cc0003b9f23f3a9c3b366ce2c8de9d2d943a474d306 honeytail-1.8.2-1.x86_64.rpm' | sha256sum -c
Install the package.
sudo rpm -i honeytail-1.8.2-1.x86_64.rpm
The packages install honeytail, its config file /etc/honeytail/honeytail.conf,
and some start scripts.
Build honeytail from source if you need it in an unpackaged form or for ad-hoc use.
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-linux-amd64
Verify the binary.
echo 'd774112265ee8e98c6221232461cf36c35faf844005cc98b43b55bb375761766 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-linux-arm64
Verify the binary.
echo 'c5a57a729b0ccf4ca0f2287c862538812604f5fd67d102372e91215701afdbe1 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Download the 1.8.2 binary.
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.8.2/honeytail-darwin-amd64
Verify the binary.
echo '49c976e9b98fa238c265ec6afce35d41767a3f8d67defa547bff933485f40a83 honeytail' | shasum -a 256 -c
Set the permissions to allow execution.
chmod 755 ./honeytail
Clone the Honeytail repository.
git clone https://github.com/honeycombio/honeytail
Install from source.
cd honeytail; go install
You should modify the config file and uncomment and set:
ParserName to jsonWriteKey to your API key, available from the account pageLogFiles to the path for the log file you want to ingest, or - for stdinDataset to the name of the dataset you wish to create with this log file. Start up a honeytail process using upstart or systemd or by launching the process by hand.
This will tail the log file specified in the config and leave the process running as a daemon.
sudo initctl start honeytail
sudo systemctl start honeytail
honeytail -c /etc/honeytail/honeytail.conf
To backfill existing data, run honeytail with --backfill the first time:
honeytail -c /etc/honeytail/honeytail.conf \
--file /var/log/myapp/log12.json \
--backfill
This command can also be used at any point to backfill from older, rotated log files. You can read more about our backfill behavior here.
Note: (If you have chosen to backfill from old JSON logs, do not forget to transition into the default streaming behavior to stream live logs to Honeycomb!)
Honeycomb expects all events to contain a timestamp field; if one is not provided, the server will associate the current time of ingest with the given payload.
By default, we look for a few candidate fields based on name ("timestamp", "time") and handle the following time formats:
2006-01-02T15:04:05Z07:00)2006-01-02T15:04:05.999999999Z07:00)Mon Jan 2 15:04:05 MST 2006)Mon Jan 02 15:04:05 -0700 2006)2006-01-02 15:04:05.999999999 -0700 MST)If your timestamps are not correctly handled by the above formats,
use the --json.timefield and --json.format flags
to help honeytail understand where and how to extract the event’s timestamp.
For example, given a JSON log file with events like the following:
{"color":"orange","size":3,"server_time":"Aug 12 2016, 15:12:06 -0800"}
{"color":"blue","server_time":"Sep 01 2016, 06:10:32 -0800","size":4}
The command to consume those log lines (while retaining the "server_time" field as the event’s timestamp)
would look something like:
honeytail --writekey=YOUR_API_KEY --dataset="API Server Logs" --parser=json \
--file=/var/log/api_server.log \
--json.timefield="server_time" --json.format="%b %d %Y, %k:%M:%S %z"
The --json.timefield="server_time" argument tells honeytail to consider the "server_time" value
to be the canonical timestamp for the events in the specified file.
The --json.format argument specifies the timestamp format to be used while parsing.
(It understands common strftime formats.)
Ultimately, the above command would would produce events with the fields (note the times below are represented in UTC; Honeycomb parses time zone information if provided).
| time | color | size |
|---|---|---|
| 2016-08-12T23:12:06Z | orange | 3 |
| 2016-09-01T14:10:32Z | blue | 4 |
Did you find what you were looking for?