honeycloudfront to support ingestion of AWS CloudFront access
These logs are useful for visualizing questions such as “How many cache misses are happening
in CloudFront?" or “How much bandwidth is CloudFront saving us?".
The source is available on Github and instructions for getting started are provided here.
This implementation supports Web Distributions. RTMP distributions are not supported. If you require RTMP support, please file an issue and CC the maintainers.
honeycloudfront assumes access to an AWS access key ID and AWS secret access
key with the proper permissions. It will attempt to obtain these via the default
~/.aws/config, by the proper environment variables, or by an IAM
EC2 instance profile. See the AWS guide on providing
for more details.
See the provided IAM policy
JSON in the
honeyaws repository for one example of a policy which has the proper
permissions. This can be scoped down to more specific resources if desired.
honeycloudfront is available as part of the Honeycomb AWS
Bundle or as a standalone binary.
honeycloudfront, use the following instructions:
wget -q https://honeycomb.io/download/honeyaws/v1.4.0/honeyaws_1.4.0_amd64.deb && \ echo 'b15409f56e67e0058f2766554c25b441abadc386cb2fc0ce41f385b4be909e09 honeyaws_1.4.0_amd64.deb' | sha256sum -c && \ sudo dpkg -i honeyaws_1.4.0_amd64.deb
wget -q https://honeycomb.io/download/honeyaws/v1.4.0/honeyaws-1.4.0-1.x86_64.rpm && \ echo '1abce68c81380e3facc53ed8cb018b0980d6e906ace5ca8b403a251130e582e0 honeyaws-1.4.0-1.x86_64.rpm' | sha256sum -c && \ sudo rpm -i honeyaws-1.4.0-1.x86_64.rpm
wget -q -O honeycloudfront https://honeycomb.io/download/honeyaws/v1.4.0/honeycloudfront-linux-amd64 && \ echo '5ba3bc3c9972f45ada7a9bf82dcd27ac05dc0af75a7361a28178478909c8a648 honeycloudfront' | sha256sum -c && \ chmod 755 ./honeycloudfront
wget -q -O honeycloudfront https://honeycomb.io/download/honeyaws/v1.4.0/honeycloudfront-linux-arm64 && \ echo 'e19f9a3b00173fa2affec410c240bf7d10405a90d5fd5e046b74a969489b8b52 honeycloudfront' | sha256sum -c && \ chmod 755 ./honeycloudfront
wget -q -O honeycloudfront https://honeycomb.io/download/honeyaws/v1.4.0/honeycloudfront-darwin-amd64 && \ echo 'b9755bc0528a82d9b4b41d71121bdb349605f2a9ee38aa6512aaab0fc6f36a99 honeycloudfront' | shasum -a 256 -c && \ chmod 755 ./honeycloudfront
honeycloudfront interactively (for beginning exploration,
debugging credential management, etc) or as a daemon. Try running some commands
interactively at first to get a feel for using the tool and then configure it to
run as a proper system service when you’re ready to be ingesting continuously.
To show all distributions, invoke
$ honeycloudfront ls EVDBLY2TVIYCVB D11111ABCDEF8Q S11A16G5KZMEQD
To ingest access logs from a distribution, use
honeycloudfront ingest with
one or more distribution names. Set your Honeycomb write
key with the
--writekey flag. By default the
events will be sent to a dataset called
Note: If access logs are not configured for the distribution it will throw an error. Please see the documentation on CloudFront access logs to learn how to enable this feature.
e.g. Ingesting logs from one distribution with ID
$ honeycloudfront --writekey=YOUR_API_KEY ingest S11A16G5KZMEQD ...
Ingesting logs from multiple specific distributions:
$ honeycloudfront --writekey=YOUR_API_KEY ingest EVDBLY2TVIYCVB D11111ABCDEF8Q S11A16G5KZMEQD ...
honeycloudfront ingest without any arguments will use all available
(“described”) distributions in your configured AWS region. With arguments, it
will ingest logs for the specified load balancer names.
$ honeycloudfront --writekey=YOUR_API_KEY ingest ...ingesting logs from all distributions...
By default, the agent will drop state files (to avoid sending duplicate events) in the
current working directory where it is invoked. To change where these
files are kept, use the
Sampling is a great way to send fewer events (thereby keeping more history and
reducing costs) while still preserving most relevant information. To set a
sample rate while using one of the Honeycomb AWS tools, use the
flag. While the tools run, this base rate will be automatically adjusted by the
Honeycomb AWS tools using dynamic sampling to keep more interesting traffic at a
For instance, setting the sample flag to 20 will send 1 out of every 20 requests
processed to Honeycomb by default. Fields such as
elb_status_code are used to
lower this ratio for rarer, but relevant, events such as HTTP 500-level errors.
$ honeyelb --samplerate 20 ... ingest ...
honeycloudfront, while supporting a interactive workflow for initial discovery and
experimentation, is meant to be invoked as a long-running process by a system
To do this, edit the system init files (Upstart and systemd are supported) installed by the package manager to add the API key.
Once you receive data from
honeycloudfront you will want to explore it. The
descriptions of the sent fields is available in the AWS documentation for Web
Here are some suggestions for things to try:
COUNTto see which and how many requests were served with cache status
MAX(time_taken)to see which URIs took the longest to serve
MAX(sc_bytes)to see when the largest requests were served and how big they were in bytes