> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Send NGINX Logs
> Parse NGINX logs with Honeytail and send them to Honeycomb to analyze web traffic, request patterns, and service activity across your infrastructure.
export const honeytail = {
"version": "1.10.0",
"deb_amd64": {
"sha256": "3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb"
},
"deb_arm64": {
"sha256": "4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb"
},
"rpm": {
"sha256": "b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-1.10.0-1.x86_64.rpm"
},
"bin_linux_amd64": {
"sha256": "c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64"
},
"bin_linux_arm64": {
"sha256": "1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64"
},
"bin_darwin_amd64": {
"sha256": "9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d",
"url": "https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64"
},
"bin_name": "honeytail"
};
NGINX is one of the most popular web servers today.
In a world driven by the web and connected APIs, its logs are a great candidate for surfacing a birds' eye view of activity in your service.
Explore the [Honeytail NGINX Example App](https://github.com/honeycombio/honeytail/tree/main/examples/nginx) and read the [blog post with user tips on sending NGINX logs to Honeycomb](https://www.honeycomb.io/blog/sending-nginx-logs-honeycomb).
## Setup
Capturing web logs for Honeycomb requires:
1. installing our agent, `honeytail`
2. configuring it to parse your NGINX logs correctly
3. launching honeytail
### Install the Agent
Download and install the latest `honeytail` by running:
Download the `honeytail_1.10.0_amd64.deb` package.
```shell theme={}
wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb
```
Verify the package.
```shell theme={}
echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c
```
Install the package.
```shell theme={}
sudo dpkg -i honeytail_1.10.0_amd64.deb
```
The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
and some start scripts.
Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
Download the `honeytail_1.10.0_arm64.deb` package.
```shell theme={}
wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb
```
Verify the package.
```shell theme={}
echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c
```
Install the package.
```shell theme={}
sudo dpkg -i honeytail_1.10.0_arm64.deb
```
The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
and some start scripts.
Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
Download the `honeytail_1.10.0-1.x86_64.rpm` package.
```shell theme={}
wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm
```
Verify the package.
```shell theme={}
echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c
```
Install the package.
```shell theme={}
sudo rpm -i honeytail_1.10.0-1.x86_64.rpm
```
The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`,
and some start scripts.
Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use.
Download the 1.10.0 binary.
```shell theme={}
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64
```
Verify the binary.
```shell theme={}
echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c
```
Set the permissions to allow execution.
```shell theme={}
chmod 755 ./honeytail
```
Download the 1.10.0 binary.
```shell theme={}
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64
```
Verify the binary.
```shell theme={}
echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c
```
Set the permissions to allow execution.
```shell theme={}
chmod 755 ./honeytail
```
Download the 1.10.0 binary.
```shell theme={}
wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64
```
Verify the binary.
```shell theme={}
echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c
```
Set the permissions to allow execution.
```shell theme={}
chmod 755 ./honeytail
```
Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository.
```shell theme={}
git clone https://github.com/honeycombio/honeytail
```
Install from source.
```shell theme={}
cd honeytail; go install
```
You should modify the config file and uncomment and set:
* `ParserName` to `nginx`
* `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account)
* `LogFiles` to the path for the log file you want to ingest.
For NGINX, this is typically `/var/log/nginx/access.log`.
* `Dataset` to the name of the dataset you wish to create with this log file.
Honeytail also supports [configuration with YAML](/send-data/logs/structured/honeytail/#convert-command-lines-to-yaml-configuration).
### Identify Log Locations + Formats
Make sure to run through [Optional Configuration](#optional-configuration) below before running `honeytail`, in order to get the richest metadata out of your web traffic and into your logs.
In addition to the standard configuration captured in `/etc/honeytail/honeytail.conf`, you will want to set the two options in the `Nginx Parser Options` section:
* `ConfigFile`: the path to your NGINX config file: whichever part of it contains the definition for the log format
* `LogFormatName`: the name of the log format used to produce the NGINX access log file
For example, if your nginx config file is at `/etc/nginx/nginx.conf` and has the following snippet:
```nginx theme={}
log_format my_favorite_format '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent';
access_log /var/log/nginx/access.log my_favorite_format;
```
... then `ConfigFile` should be set to `/etc/nginx/nginx.conf` and your `LogFormatName` value should be set to `my_favorite_format`.
Or configure `honeytail` to read the nginx logs using command line parameters:
```shell theme={}
honeytail \
--parser=nginx \
--dataset=examples.honeytail-nginx \
--writekey=$HONEYCOMB_WRITE_KEY \
--nginx.conf=/etc/nginx/nginx.conf \
--nginx.format=my_favorite_format \
--file=/var/log/nginx/access.log
```
### Launch the Agent
Start up a `honeytail` process using `upstart` or `systemd` or by launching the process by hand.
```shell theme={}
sudo initctl start honeytail
```
```shell theme={}
sudo systemctl start honeytail
```
```shell theme={}
honeytail -c /etc/honeytail/honeytail.conf
```
## Backfilling Archived Logs
In addition to getting current logs flowing, you can backfill old logs into Honeycomb to kickstart your dataset.
By running `honeytail` from the command line, you can import old logs separate from tailing your current logs.
Adding the `--backfill` flag to `honeytail` adjusts a number of settings to make it appropriate for backfilling old data, such as stopping when it gets to the end of the log file instead of the default behavior of waiting for new content (like `tail`).
The specific locations on your system may vary from ours, but once you fill in your system's values instead of our examples, you can backfill using this command:
```shell theme={}
honeytail --writekey=YOUR_API_KEY --dataset="nginx API logs" --parser=nginx \
--file=/var/log/nginx/access.16.log \
--nginx.conf=/etc/nginx/nginx.conf \
--nginx.format=api_fmt \
--backfill
```
This command can be used at any point to backfill from archived log files.
You can read more about our [agent honeytail](/send-data/logs/structured/honeytail/) or its [backfill behavior](/send-data/logs/structured/honeytail/#backfilling-existing-data) here.
`honeytail` does not unzip log files, so you will need to do this before backfilling.
Easiest way—pipe to STDIN: `zcat *.gz | honeytail --file - --backfill --all-the-other-flags.`
## Troubleshooting
Check out [`honeytail` Troubleshooting](/troubleshoot/common-issues/sending-data/#honeytail) for debugging tips.
## Optional Configuration
Nginx logs can be an incredibly powerful, high-level view of your system—especially so if they are configured correctly and enriched with custom, application-specific information about each request.
Below are two simple ways to pack those logs with more useful metadata.
### Missing Default Options
Nginx comes with some fairly powerful optional log fields that are not included by default.
This is the `log_format` we recommend for any configuration file (note the extra quotes around some fields):
```nginx theme={}
log_format combined '$remote_addr - $remote_user [$time_local] $host '
'"$request" $status $bytes_sent $body_bytes_sent $request_time '
'"$http_referer" "$http_user_agent" $request_length "$http_authorization" '
'"$http_x_forwarded_proto" "$http_x_forwarded_for" $server_name';
access_log /var/log/nginx/access.log combined;
```
You may already have an `access_log` line, but by defining a `log_format` (`combined`, in the example above) and specifying the format name (`--nginx.format=combined`), you will be able to take advantage of all of these additional fields.
Make sure that all fields that start `$http_` are quoted in your `log_format`:
* `$bytes_sent`: the size of the response sent back to the client, including headers
* `$host`: the requested Host header, identifying how your server was addressed
* `$http_authorization`: authorization headers, for associating logs with individual users (must be quoted)
* `$http_referer`: the referring site, if the client followed a link to your site (must be quoted)
* `$http_user_agent`: the User-Agent header, useful in identifying your clients (must be quoted)
* `$http_x_forwarded_for`: the origin IP address, if running behind a load balancer (must be quoted)
* `$http_x_forwarded_proto`: the origin protocol, if terminating TLS in front of nginx (must be quoted)
* `$remote_addr`: the IP address of the host making the connection to nginx
* `$remote_user`: the user name supplied when/if using basic authentication
* `$request_id`: an nginx-generated unique ID to every request (only available in nginx version 1.11 and later).
* `$request_length`: the length of the client's request, including headers and body
* `$request_time`: the time (in ms) the server took to respond to the request
* `$request`: the HTTP method, request path, and protocol version
* `$server_name`: the hostname of the machine accepting the request
* `$status`: the HTTP status code returned for this request
### Embedding Custom Response Headers
Nginx can also be configured to extract custom request and response headers.
Of the two, response headers are the most powerful in this case—they can carry application-specific IDs or timers back through to the nginx log.
Having all of the information pertinent to a single request, available in a single log line, can be an incredibly powerful tool in diagnosing the origin of a problem in your system.
To include a specific response header in your `access.log`, add an [`$upstream_http_`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_http_) variable to your `log_format`—the response header values will be written out and ingested by our nginx parser!
Make sure to put quotes around these variables to capture any embedded spaces.
For example, an `X-RateLimit-Remaining` header can be output by adding `$upstream_http_x_ratelimit_remaining` to the `log_format` line.
See the nginx docs for more about [extracting metadata from the HTTP response](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables) or [request](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_http_).
As with other fields which may output strings (for example,`$http_user_agent`), be careful when logging strings—add an extra set of double quotes around values which might contain spaces, in order to ensure correct parsing.
**A final trick**: sometimes, response headers may be set for logging that should not be exposed back to the user.
In this case, the [`proxy_hide_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive may be used to strip out specific headers by name:
```nginx theme={}
log_format combined `... "$upstream_x_internal_top_secret" ...`; # Wrap string values with double quotes
access_log /var/log/nginx/access.log combined;
location / {
proxy_pass http://127.0.0.1:8080; # Expose port 8080
proxy_hide_header X-Internal-Top-Secret; # Strip from client
}
```
## Scrubbing Personally Identifiable Information
While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data which may contain sensitive user information.
With that in mind, we recommend using `honeytail`'s nginx parser, but adding a `--scrub_field=sensitive_field_name` flag to hash the concrete `sensitive_field_name` value, or `--drop_field=sensitive_field_name` to drop it altogether and prevent it being sent to Honeycomb's servers.
Find [more information about dropping or scrubbing sensitive field](/send-data/logs/structured/honeytail/#dropping-or-scrubbing-fields).
## Parsing URL Patterns
`honeytail` can break URLs up into their component parts, storing extra information in additional columns.
This behavior is turned on by default for the `request` field on `nginx` datasets, but can become more useful with a little bit of guidance from you.
See [`honeytail`'s documentation](/send-data/logs/structured/honeytail/#parsing-url-patterns) for details on configuring our agent to parse URL strings.
## Open Source
[Honeytail](https://github.com/honeycombio/honeytail) is open source and Apache 2.0 licensed.