To ingest AWS Elastic Load Balancer access
logs or AWS Application Load Balancer access
(useful for visualizing questions such as “Which backend servers are taking the
longest to answer requests?" or “Which calls to our app are returning non-200
HTTP status codes?") Honeycomb provides a tool called
The source is available on Github and instructions for getting started are provided here.
Note: To ingest Application Load Balancer (ALB) logs, substitute
honeyelb in the instructions here.
Please use the following instructions to install
honeyelb. It is available as
part of the Honeycomb AWS Bundle or
as a standalone binary.
wget -q https://honeycomb.io/download/honeyaws/v1.4.1/honeyaws_1.4.1_amd64.deb && \ echo 'edd731c956436e2b7a3bf691d35bd4f44c66e476a47b864aa472549b0b3823d4 honeyaws_1.4.1_amd64.deb' | sha256sum -c && \ sudo dpkg -i honeyaws_1.4.1_amd64.deb
wget -q https://honeycomb.io/download/honeyaws/v1.4.1/honeyaws-1.4.1-1.x86_64.rpm && \ echo 'b8741fd63ee35109e307fd230c153ecddafe4fc8c00514ea61dba3699c4edeaa honeyaws-1.4.1-1.x86_64.rpm' | sha256sum -c && \ sudo rpm -i honeyaws-1.4.1-1.x86_64.rpm
wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.1/honeyelb-linux-amd64 && \ echo '959cf009ca8135df19b52f9a2a8fc72fa5d4bf52c0d35b03a78a176e0fc4453d honeyelb' | sha256sum -c && \ chmod 755 ./honeyelb
wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.1/honeyelb-linux-arm64 && \ echo 'a1c0d00bc4529d340006a2c2b6a6bdb7721fd6fbaf986caa202684eba34dbef7 honeyelb' | sha256sum -c && \ chmod 755 ./honeyelb
wget -q -O honeyelb https://honeycomb.io/download/honeyaws/v1.4.1/honeyelb-darwin-amd64 && \ echo '7e51cb8919993aaab7a2655ec3f037b6cd2bdbf8781e1e918fc512c7553762a2 honeyelb' | shasum -a 256 -c && \ chmod 755 ./honeyelb
honeyelb assumes access to an AWS access key ID and AWS secret access key with
the proper permissions. It will attempt to obtain these via the default profile
~/.aws/config, by the proper environment variables, or by an IAM EC2
instance profile. See the AWS guide on providing
for more details.
See the provided IAM policy
JSON in the
honeyelb repository for one example of a policy which has the proper
permissions. This can be scoped down to more specific resources if desired.
honeyelb can be used interactively (meant for beginning exploration,
debugging credential management, etc.) or as a daemon. Try running some
commands interactively at first to get a feel for using the tool and then
configure it to run as a proper system service when you’re ready to be
To show all ELBs, you can invoke
$ honeyelb ls frontend internal-service service-proxy
To ingest access logs from an ELB, use
honeyelb ingest with one or more ELB
--writekey flag must be set to your API key.
By default the events will be sent to a dataset called
Note: If access logs are not configured for the ELB it will throw an error. Please see enable access logs for your Classic Load Balancers to enable this feature.
e.g. Ingesting logs from one ELB named
$ honeyelb --writekey=YOUR_API_KEY \ ingest frontend ...
Ingesting logs from multiple specific load balancers (named
$ honeyelb --writekey=YOUR_API_KEY \ ingest frontend internal-service service-proxy ...
honeyelb ingest without any arguments will use all available (“described”)
load balancers in your configured AWS region. With arguments, it will ingest
logs for the specified load balancer names.
$ honeyelb --writekey=YOUR_API_KEY \ ingest ...ingesting logs from all LBs in DescribeLoadBalancers...
A Honeycomb dataset is created automatically upon receiving the first events
honeyelb. The dataset name can be specified using the
--dataset= flag. The default dataset name is
aws-elb-access for ELB and
aws-alb-access for ALB.
The agent will drop state files (to avoid sending duplicate events) in the
current working directory where it is invoked by default. To modify where these
files are kept, use the
Sampling is a great way to send fewer events (thereby keeping more history and
reducing costs) while still preserving most relevant information. To set a
sample rate while using one of the Honeycomb AWS tools, use the
flag. While the tools run, this base rate will be automatically adjusted by the
Honeycomb AWS tools using dynamic sampling to keep more interesting traffic at a
For instance, setting the sample flag to 20 will send 1 out of every 20 requests
processed to Honeycomb by default. Fields such as
elb_status_code are used to
lower this ratio for rarer, but relevant, events such as HTTP 500-level errors.
$ honeyelb --samplerate 20 ... ingest ...
honeyelb, while supporting a interactive workflow for initial discovery and
experimentation, is meant to be invoked as a long-running process by a system
To do this, edit the system init files (Upstart and systemd are supported) installed by the package manager to add the API key.
If you’re not seeing your data in Honeycomb, please try the following:
Once you receive data from
honeyelb you will want to explore it. The
descriptions of the sent fields is available in the AWS documentation for ELB
There is one small difference: the
backend:port keys from
that guide are represented as
backend_authority in the
Honeycomb events. The listed fields for ALB are
Here are some suggestions for things to try:
MAX(backend_processing_time)to see which server(s) answered your slowest requests
P99(backend_processing_time)to see which endpoints (URL paths) take the longest
P99(backend_processing_time)to see which ELBs returned the most HTTP status code 200, 404, 500 etc. responses
MIN()to see when backends have timed out (timeouts are represented by -1 response time in the ELB access logs)
trace_id contains <your-parent-trace-id>