> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Send Unstructured Logs with Honeytail + Custom Regex

> Parse unstructured log files using custom regular expressions and Honeytail, then send the parsed events to Honeycomb for high-cardinality querying.

export const honeytail = {
  "version": "1.10.0",
  "deb_amd64": {
    "sha256": "3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb"
  },
  "deb_arm64": {
    "sha256": "4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb"
  },
  "rpm": {
    "sha256": "b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-1.10.0-1.x86_64.rpm"
  },
  "bin_linux_amd64": {
    "sha256": "c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64"
  },
  "bin_linux_arm64": {
    "sha256": "1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64"
  },
  "bin_darwin_amd64": {
    "sha256": "9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d",
    "url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64"
  },
  "bin_name": "honeytail"
};

Use custom regular expressions and Honeytail, our lightweight tool to tail your existing log files, parse the content, and send it to Honeycomb.

## Installation

Download and install the latest `honeytail` by running:

<Tabs>
  <Tab title="deb-amd64">
    Download the `honeytail_1.10.0_amd64.deb` package.

    ```shell theme={}
    wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb
    ```

    Verify the package.

    ```shell theme={}
    echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917  honeytail_1.10.0_amd64.deb' | sha256sum -c
    ```

    Install the package.

    ```shell theme={}
    sudo dpkg -i honeytail_1.10.0_amd64.deb
    ```

    The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
    and some start scripts.
    Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
  </Tab>

  <Tab title="deb-arm64">
    Download the `honeytail_1.10.0_arm64.deb` package.

    ```shell theme={}
    wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb
    ```

    Verify the package.

    ```shell theme={}
    echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed  honeytail_1.10.0_arm64.deb' | sha256sum -c
    ```

    Install the package.

    ```shell theme={}
    sudo dpkg -i honeytail_1.10.0_arm64.deb
    ```

    The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
    and some start scripts.
    Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
  </Tab>

  <Tab title="rpm">
    Download the `honeytail_1.10.0-1.x86_64.rpm` package.

    ```shell theme={}
    wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm
    ```

    Verify the package.

    ```shell theme={}
    echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863  honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c
    ```

    Install the package.

    ```shell theme={}
    sudo rpm -i honeytail_1.10.0-1.x86_64.rpm
    ```

    The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
    and some start scripts.
    Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
  </Tab>

  <Tab title="bin-linux-amd64">
    Download the 1.10.0 binary.

    ```shell theme={}
    wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64
    ```

    Verify the binary.

    ```shell theme={}
    echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5  honeytail' | shasum -a 256 -c
    ```

    Set the permissions to allow execution.

    ```shell theme={}
    chmod 755 ./honeytail
    ```
  </Tab>

  <Tab title="bin-linux-arm64">
    Download the 1.10.0 binary.

    ```shell theme={}
    wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64
    ```

    Verify the binary.

    ```shell theme={}
    echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa  honeytail' | shasum -a 256 -c
    ```

    Set the permissions to allow execution.

    ```shell theme={}
    chmod 755 ./honeytail
    ```
  </Tab>

  <Tab title="bin-darwin-amd64">
    Download the 1.10.0 binary.

    ```shell theme={}
    wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64
    ```

    Verify the binary.

    ```shell theme={}
    echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d  honeytail' | shasum -a 256 -c
    ```

    Set the permissions to allow execution.

    ```shell theme={}
    chmod 755 ./honeytail
    ```
  </Tab>

  <Tab title="source">
    Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository.

    ```shell theme={}
    git clone https://github.com/honeycombio/honeytail
    ```

    Install from source.

    ```shell theme={}
    cd honeytail; go install
    ```
  </Tab>
</Tabs>

You should modify the config file and uncomment and set:

* `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account)
* `LogFiles` to the path for the log file you want to ingest, or `-` for stdin
* `Dataset` to the name of the dataset you wish to create with this log file.
* `ParserName` to `regex`
* `LineRegex` to a regular expression with named capture groups.

## Launch the Agent

Start up a `honeytail` process using `upstart` or `systemd` or by launching the process by hand.
This will tail the log file specified in the config and leave the process running as a daemon.

<Tabs>
  <Tab title="upstart">
    ```shell theme={}
    sudo initctl start honeytail
    ```
  </Tab>

  <Tab title="systemd">
    ```shell theme={}
    sudo systemctl start honeytail
    ```
  </Tab>

  <Tab title="manual">
    ```shell theme={}
    honeytail -c /etc/honeytail/honeytail.conf
    ```
  </Tab>
</Tabs>

## Backfilling Archived Logs

To backfill existing data, run `honeytail` with `--backfill` the first time:

```shell theme={}
honeytail -c /etc/honeytail/honeytail.conf \
  --file /var/log/myapp/log12.log \
  --backfill
```

This command can also be used at any point to backfill from older, rotated log files.
You can read more about our [backfill behavior here](/send-data/logs/structured/honeytail/).

<Note>
  If you have chosen to backfill from old logs, do not forget to transition into the default streaming behavior to stream live logs to Honeycomb!
</Note>

## Regexes

We use golang's [regexp](https://golang.org/pkg/regexp/) package, which uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax).

### Specifying Regexes

Command line: use the **`--regex.line_regex`** flag to tell `honeytail` how to extract data from a log line.

You must provide at least one regex.
You may optionally specify multiple regexes.
Lines will be parsed by the first regex to find a match.
Precedence is based on the order you pass in `line_regex`, so specify your regexes from most-specific to least-specific.

On the command line, you will need to wrap the regex in quotes.

```shell theme={}
honeytail \
    --writekey YOUR_API_KEY \
    --file PATH/FILE.LOG \
    --parser regex \
    --dataset "MY_TEST_SET" \
    --backfill \
    --regex.line_regex="\[(?P<time>\d{2}:\d{2}:\d{2})\] (?P<message>\w+)" \
    --regex.line_regex="(?P<field1>\w+) (?P<field2>\w+)"
```

Equivalent configuration file specification.
Note that you should not wrap the regex in quotes here.

```toml theme={}
[Regex Parser Options]
; a regular expression with named capture groups representing the fields you want parsed
LineRegex = \[(?P<time>\d{2}:\d{2}:\d{2})\] (?P<message>\w+)
LineRegex = (?P<field1>\w+) (?P<field2>\w+)
```

### Regex Syntax

Regexes must contain at least one named capture group.
Use the `(?P<name>re)` syntax for named groups. Example:

Log file

```log theme={}
[2017/11/07 22:59:46] 200 ...
[2017/11/07 22:59:48] 500 ...
[2017/11/07 23:01:02] 404 ...
```

with

```bash theme={}
--regex.line_regex="\[(?P<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\] (?P<status>\d+)"
```

will yield rows like this:

```json theme={}
{
  time: "2017/11/07 22:59:46",
  status: "200"
}
```

### Nested Regex Grouping

Nested groups are supported.
For example:

```bash theme={}
--regex.line_regex="(?P<outer>[^ ]* (?P<inner1>[^ ]*) (?P<inner2>[^ ]*))"
```

will parse a log line "`A B C`" into `{ outer: "A B C", inner1: "B", inner2: "C" }`.

## Timestamp Parsing

Honeycomb expects all events to contain a timestamp field; if one is not provided, the server will associate the current time of ingest with the given payload.

Use the `--regex.timefield` and `--regex.time_format` flags to help `honeytail` understand where and how to extract the event's timestamp.

For example, given a log file like the following:

```log theme={}
[08/Oct/2015:00:26:26 +0000] 200 174 0.099
```

A command to consume those log lines (retaining the `"local_time"` field as the event's timestamp would look like):

```shell theme={}
honeytail \
    --parser=regex \
    --writekey=YOUR_API_KEY \
    --file=server.log  \
    --dataset='MY_DATASET' \
    --backfill \
    --regex.line_regex=SOME_REGEX \
    --regex.timefield="local_time" \
    --regex.time_format="%d/%b/%Y:%H:%M:%S %z"
```

The `--regex.timefield="local_time"` argument tells `honeytail` to consider the `"local_time"` value to be the canonical timestamp for the events in the specified file.

The `--regex.time_format` argument specifies the timestamp format to be used while parsing.
(It understands common [`strftime`](https://www.strfti.me/) formats.)