Getting AWS CloudTrail logs into Honeycomb

Honeycomb provides honeycloudtrail to support ingestions of AWS CloudTrail Log files. These logs are useful to monitor your trails and visualizing questions, such as “Which API request is this account ID making most frequently?" or “Are there specific IPs trying to access our S3 buckets?"

The source is available on Github and instructions for getting started are provided here.

Prerequisites

honeycloudtrail assumes access to an AWS access key ID and AWS secret access key with the proper permissions. It will attempt to obtain these via the default profile in ~/.aws/config, by the proper environment variables, or by an IAM EC2 instance profile. See the AWS guide on providing credentials for more details.

See the provided IAM policy JSON in the honeyaws repository for one example of a policy which has the proper permissions. This can be scoped down to more specific resources if desired.

Install

Use the following instructions to install honeycloudtrail. It is available as part of the Honeycomb AWS Bundle or as a standalone binary.

deb

wget -q https://honeycomb.io/download/honeyaws/v1.4.3/honeyaws_1.4.3_amd64.deb && \
      echo 'c2dd79fec9e2346568de562dd790fd1e474a62fed3a6d3d21e8f1b3472b03418  honeyaws_1.4.3_amd64.deb' | sha256sum -c && \
      sudo dpkg -i honeyaws_1.4.3_amd64.deb

wget -q https://honeycomb.io/download/honeyaws/v1.4.3/honeyaws-1.4.3-1.x86_64.rpm && \
      echo '98d60d2b898f4b7a03abed013b21c6f2d3a86014cc08d2dbbd78e51c3b8d9dcd  honeyaws-1.4.3-1.x86_64.rpm' | sha256sum -c && \
      sudo rpm -i honeyaws-1.4.3-1.x86_64.rpm

wget -q -O honeycloudtrail https://honeycomb.io/download/honeyaws/v1.4.3/honeycloudtrail-linux-amd64 && \
      echo '8de18f3750b88d162b9ac36746a628ecd1b55d614b8b4da149188bfe01774f4f  honeycloudtrail' | sha256sum -c && \
      chmod 755 ./honeycloudtrail

wget -q -O honeycloudtrail https://honeycomb.io/download/honeyaws/v1.4.3/honeycloudtrail-linux-arm64 && \
      echo '82d3d97d63b9f2c664f89b62c91e6842f11c05e37bdd0cba5f12cf76b8d1d8dc  honeycloudtrail' | sha256sum -c && \
      chmod 755 ./honeycloudtrail

wget -q -O honeycloudtrail https://honeycomb.io/download/honeyaws/v1.4.3/honeycloudtrail-darwin-amd64 && \
      echo 'fbea139d2b6b32f7436b9effe8589b7972f7db163393fc3377a9636cee5ffaa6  honeycloudtrail' | shasum -a 256 -c && \
      chmod 755 ./honeycloudtrail

Usage

Use honeycloudtrail interactively (for beginning exploration, debugging credential management) or as a daemon. Try running some commands interactively at first to get a feel for using the tool and then configure it to run as a proper system service when you are ready to be ingesting continuously.

Interactive

To show all trails, invoke honeycloudtrail ls:

$ honeycloudtrail ls
s3-trail
elb-frontend-trail
service-trail
...

To ingest access logs from a distribution, use honeycloudtrail ingest with one or more distribution names. Set your Honeycomb write key with the --writekey flag. By default the events will be sent to a dataset called aws-cloudtrail-access.

Note: If an S3 bucket is not configured for the trail it will throw an error. See the documentation on CloudTrail to learn how to enable S3 bucket logging.

For example, ingesting logs from one trail named s3-trail:

honeycloudtrail --writekey=YOUR_API_KEY \
  ingest s3-trail
...

Ingesting logs from multiple specific load balancers (named s3-trail, elb-frontend-trail, and service-trail):

honeycloudtrail --writekey=YOUR_API_KEY \
  ingest s3-trail elb-frontend-trail service-trail
...

honeycloudtrail ingest without any arguments will ingest all available CloudTrail trails in your configured AWS region. With arguments, it will ingest logs for the specified trail names.

Sampling

Sampling is a great way to send fewer events (thereby keeping more history and reducing costs) while still preserving most relevant information. To set a sample rate while using one of the Honeycomb AWS tools, use the --samplerate flag. While the tools run, this base rate will be automatically adjusted by the Honeycomb AWS tools using dynamic sampling to keep more interesting traffic at a higher rate.

For instance, setting the sample flag to 20 will send 1 out of every 20 requests processed to Honeycomb by default. Fields such as elb_status_code are used to lower this ratio for rarer, but relevant, events such as HTTP 500-level errors.

honeyelb --samplerate 20 ...  ingest ...

Daemon

honeycloudtrail, while supporting a interactive workflow for initial discovery and experimentation, is meant to be invoked as a long-running process by a system service manager.

To do this, edit the system init files (Upstart and systemd are supported) installed by the package manager to add the API key.

Once you receive data from honeycloudtrail you will want to explore it. The descriptions of the sent fields is available in the AWS documentation for CloudTrail logs.

Here are some suggestions for things to try:

GROUP BY EventSource and COUNT_DISTINCT(SourceIPAddress) to see which IPs are using which services.
GROUP BY ARN to see which IAM roles are being used the most frequently
GROUP BY EventName to see the most commonly used API calls

Did you find what you were looking for?

Additional Feedback (Optional)

If you have questions, join our Pollinators Community Slack, or for Pro/Enterprise users, contact Support at support@honeycomb.io.

Honeycomb.io Docs