How Honeycomb's AWS Integrations Work

This section describes the mechanics of Honeycomb AWS Integrations that collect logs and metrics. It describes various collection methods and documents which AWS services use those methods. For more details, visit the Honeycomb AWS Integrations GitHub repository for Terraform or CloudFormation.

​

AWS CloudWatch Logs

Many AWS services publish logs to CloudWatch Logs. By default, most AWS services send unstructured logs to CloudWatch. Because Honeycomb is designed to work best with structured events, Honeycomb AWS Integrations convert some of those CloudWatch log streams into structured data. Deploy the Honeycomb CloudWatch Logs integration with either Terraform or CloudFormation.

​

How AWS CloudWatch Logs Integrations Work

AWS CloudWatch provides Subscription Filters, which can be configured to forward all logs in a log group to a Kinesis Firehose destination that can then stream the logs to our Honeycomb Kinesis endpoint. AWS services listed in the unstructured data column of the table below use a workflow, as shown by the diagram below, to send AWS CloudWatch logs to Honeycomb.

1. AWS CloudWatch log groups point at AWS Kinesis Firehose.
2. Firehose forwards data to Honeycomb.

Kinesis Data Firehose can invoke Lambda functions to transform incoming source data before delivering the transformed data to its destination. Supported AWS services listed in the structured data column of the table below are either JSON-formatted by default or use an additional step in their workflow, as shown by the diagram above, to structure their log data before sending it to Honeycomb.

​

Supported AWS Services Through CloudWatch Logs

Honeycomb AWS Integrations are designed to work with all services that logs to CloudWatch. For a full list of possible data sources, refer to AWS documentation for services that publish logs to CloudWatch Logs. This list is a selection of supported AWS services that publish logs through CloudWatch Logs and the type of log data Honeycomb can collect from them.

​

Working With Unstructured AWS Logs

Each field in a structured log (or event) is queryable with Honeycomb. This makes working with structured AWS log data similar to working with other types of data in Honeycomb. Unstructured logs are received by Honeycomb as events with a verbose message attribute. All unstructured log data is contained within this single attribute. For tips on working with unstructured AWS logs, refer to Work with Your AWS Data.

​

Advanced Use

The CloudWatch Logs integration is a great way to get started and gain insights across a variety of AWS services. If you want to customize the structure of the logs, you might want to write your own data transformation Lambda to be used with the Kinesis Data Firehose. Source code for our RDS data transformations can be viewed in the Agentless Integrations GitHub repository as an example.

​

AWS CloudWatch Metrics

Honeycomb AWS Integrations support all AWS services that send metrics to CloudWatch. AWS CloudWatch provides Metric Streams, which forward all metrics captured by Amazon during normal use of AWS offerings to third-party destinations, including Honeycomb. Deploy the Honeycomb Cloudwatch Metrics integration with either Terraform or CloudFormation.

​

How AWS CloudWatch Metrics Integrations work

Metrics stored in AWS CloudWatch can be streamed to other systems using AWS Kinesis Data Firehose. Honeycomb provides an endpoint that is compatible with CloudWatch Metric Streams, and stores the data it receives in a dataset for easy querying.

1. AWS Cloudwatch with Metric Streams pointed at AWS Kinesis.
2. Kinesis forwards data to Honeycomb over Data Firehose, which is configured to format data with OpenTelemetry Line Protocol (OTLP).

​

Supported AWS Services Through CloudWatch Metrics

For a full list of possible data sources, refer to AWS documentation for services that publish metrics to CloudWatch Metrics.

​

Amazon S3 Logs

Honeycomb provides an agentless integration for ingesting S3 logfiles and sending them to Honeycomb as structured events. Deploy the Honeycomb S3 Logs integration with either Terraform or CloudFormation.

​

How Amazon S3 Logs Integrations Work

AWS services listed in the the table below use a workflow, as shown by the diagram below, to send Amazon S3 logs to Honeycomb. The integration uses a Lambda function, which is subscribed to PutObject events on your bucket. The Lambda source code is available on GitHub.

1. S3 logs point at a Lambda.
2. Lambda delivers structured logs to Honeycomb.

​

Supported AWS Services Through Amazon S3 Logs

This is a complete list of AWS services logging to S3 that are supported by Honeycomb AWS Integrations.