Getting AWS Elastic Load Balancer logs into Honeycomb

To ingest AWS Elastic Load Balancer access logs or AWS Application Load Balancer access +logs (useful for visualizing questions such as “Which backend servers are taking the longest to answer requests?" or “Which calls to our app are returning non-200 HTTP status codes?") Honeycomb provides a tool called honeyelb/honeyalb.

The source is available on Github and instructions for getting started are provided here.

Note: To ingest Application Load Balancer (ALB) logs, substitute honeyalb for honeyelb in the instructions here.

Installation 🔗

Please use the following instructions to install honeyelb. It is available as part of the Honeycomb AWS Bundle or as a standalone binary.

deb

wget -q https://honeycomb.io/download/honeyaws/v1.4.0/honeyaws_1.4.0_amd64.deb && \
      echo 'b15409f56e67e0058f2766554c25b441abadc386cb2fc0ce41f385b4be909e09  honeyaws_1.4.0_amd64.deb' | sha256sum -c && \
      sudo dpkg -i honeyaws_1.4.0_amd64.deb

wget -q https://honeycomb.io/download/honeyaws/v1.4.0/honeyaws-1.4.0-1.x86_64.rpm && \
      echo '1abce68c81380e3facc53ed8cb018b0980d6e906ace5ca8b403a251130e582e0  honeyaws-1.4.0-1.x86_64.rpm' | sha256sum -c && \
      sudo rpm -i honeyaws-1.4.0-1.x86_64.rpm

wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.0/honeyelb-linux-amd64 && \
      echo '01629098f45d1ae5a59b37e73bcf7e10a96ddcc171e5dd38b726cb07f9c9c959  honeyelb' | sha256sum -c && \
      chmod 755 ./honeyelb

wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.0/honeyelb-linux-arm64 && \
      echo '02da28e1bf451051f284c0f31c5d850f0c632fdfb37f2cc4dac05df9bcbb9e41  honeyelb' | sha256sum -c && \
      chmod 755 ./honeyelb

wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.0/honeyelb-darwin-amd64 && \
      echo 'f2b5232af8b6e5bfe1ebb77e9de4fb93e2a9f7e83bf38b5e9ca6a8c60e872b3e  honeyelb' | shasum -a 256 -c && \
      chmod 755 ./honeyelb

honeyelb assumes access to an AWS access key ID and AWS secret access key with the proper permissions. It will attempt to obtain these via the default profile in ~/.aws/config, by the proper environment variables, or by an IAM EC2 instance profile. See the AWS guide on providing credentials for more details.

See the provided IAM policy JSON in the honeyelb repository for one example of a policy which has the proper permissions. This can be scoped down to more specific resources if desired.

Usage 🔗

honeyelb can be used interactively (meant for beginning exploration, debugging credential management, etc.) or as a daemon. Try running some commands interactively at first to get a feel for using the tool and then configure it to run as a proper system service when you’re ready to be ingesting continuously.

Interactive 🔗

To show all ELBs, you can invoke honeyelb ls:

$ honeyelb ls
frontend
internal-service
service-proxy

To ingest access logs from an ELB, use honeyelb ingest with one or more ELB names. The --writekey flag must be set to your API key. By default the events will be sent to a dataset called aws-elb-access.

Note: If access logs are not configured for the ELB it will throw an error. Please see enable access logs for your Classic Load Balancers to enable this feature.

e.g. Ingesting logs from one ELB named frontend:

$ honeyelb --writekey=YOUR_API_KEY \
  ingest frontend
...

Ingesting logs from multiple specific load balancers (named frontend, internal-service, and service-proxy):

$ honeyelb --writekey=YOUR_API_KEY \
  ingest frontend internal-service service-proxy
...

honeyelb ingest without any arguments will use all available (“described”) load balancers in your configured AWS region. With arguments, it will ingest logs for the specified load balancer names.

$ honeyelb --writekey=YOUR_API_KEY \
  ingest
...ingesting logs from all LBs in DescribeLoadBalancers...

A Honeycomb dataset is created automatically upon receiving the first events from honeyelb. The dataset name can be specified using the -d or --dataset= flag. The default dataset name is aws-elb-access for ELB and aws-alb-access for ALB.

The agent will drop state files (to avoid sending duplicate events) in the current working directory where it is invoked by default. To modify where these files are kept, use the --statedir flag.

Sampling 🔗

Sampling is a great way to send fewer events (thereby keeping more history and reducing costs) while still preserving most relevant information. To set a sample rate while using one of the Honeycomb AWS tools, use the --samplerate flag. While the tools run, this base rate will be automatically adjusted by the Honeycomb AWS tools using dynamic sampling to keep more interesting traffic at a higher rate.

For instance, setting the sample flag to 20 will send 1 out of every 20 requests processed to Honeycomb by default. Fields such as elb_status_code are used to lower this ratio for rarer, but relevant, events such as HTTP 500-level errors.

$ honeyelb --samplerate 20 ...  ingest ...

Daemon 🔗

honeyelb, while supporting a interactive workflow for initial discovery and experimentation, is meant to be invoked as a long-running process by a system service manager.

To do this, edit the system init files (Upstart and systemd are supported) installed by the package manager to add the API key.

Troubleshooting 🔗

If you’re not seeing your data in Honeycomb, please try the following:

Check to see if a dataset was created automatically to verify that events were sent to Honeycomb
Run honeyelb with the --debug flag

Exploration in Honeycomb 🔗

Once you receive data from honeyelb you will want to explore it. The descriptions of the sent fields is available in the AWS documentation for ELB access logs. There is one small difference: the client:port and backend:port keys from that guide are represented as client_authority and backend_authority in the Honeycomb events. The listed fields for ALB are here.

Here are some suggestions for things to try:

GROUP BY backend_authority and MAX(backend_processing_time) to see which server(s) answered your slowest requests
GROUP BY request_path and P99(backend_processing_time) to see which endpoints (URL paths) take the longest
GROUP BY elb and status_code and P99(backend_processing_time) to see which ELBs returned the most HTTP status code 200, 404, 500 etc. responses
GROUP BY backend_authority and MIN() to see when backends have timed out (timeouts are represented by -1 response time in the ELB access logs)
(ALB-only) WHERE trace_id contains <your-parent-trace-id>