This section describes the mechanics of Honeycomb AWS Integrations that collect logs and metrics. It describes various collection methods and documents which AWS services use those methods.
Many AWS services publish logs to CloudWatch Logs. By default, most AWS services send unstructured logs to CloudWatch. Because Honeycomb is designed to work best with structured events, Honeycomb AWS Integrations convert some of those CloudWatch log streams into structured data.
AWS CloudWatch provides Subscription Filters, which can be configured to forward all logs in a log group to a Kinesis Firehose destination that can then stream the logs to our Honeycomb Kinesis endpoint. AWS services listed in the unstructured data column of the table below use a workflow, as shown by the diagram below, to send AWS CloudWatch logs to Honeycomb.
Kinesis Data Firehose can invoke Lambda functions to transform incoming source data before delivering the transformed data to its destination. AWS services listed in the structured data column of the table above are either JSON-formatted by default or use an additional step in their workflow, as shown by the diagram below, to structure their log data before sending it to Honeycomb.
Honeycomb AWS Integrations are designed to work with all services that logs to CloudWatch.
For a full list of possible data sources, refer to AWS documentation for services that publish logs to CloudWatch Logs.
This list is a selection of supported AWS services that publish logs through CloudWatch Logs and the type of log data Honeycomb can collect from them.
|AWS Services supported through CloudWatch Logs||Structured||Unstructured|
|Amazon API Gateway||✔|
|AWS Elastic Beanstalk||✔|
|Amazon Elastic Container Service (ECS)||✔|
|Amazon Elastic Kubernetes Service (EKS)||✔|
|Elastic Load Balancing (ELB)||✔|
|Amazon Managed Streaming for Kafka||✔|
|AWS Network Firewall||✔|
|Amazon Relational Database Service (RDS)||✔|
|Amazon Route 53||✔|
|Amazon Simple Notification Service (SNS)||✔|
|Amazon Simple Storage Service (S3)||✔|
|AWS Step Functions||✔|
|Amazon Virtual Private Cloud (VPC)||✔|
Each field in a structured log (or event) is queryable with Honeycomb. This makes working with structured AWS log data similar to working with other types of data in Honeycomb.
Unstructured logs are received by Honeycomb as events with a verbose
All unstructured log data is contained within this single attribute.
For tips on working with unstructured AWS logs, refer to Work with Your AWS Data.
The CloudWatch Logs integration is a great way to get started and gain insights across a variety of AWS services. If you want to customize the structure of the logs, you might want to write your own data transformation Lambda to be used with the Kinesis Data Firehose. Source code for our RDS data transformations can be viewed in the Agentless Integrations GitHub repository as an example.
Honeycomb AWS Integrations support all AWS services that send metrics to CloudWatch.
AWS CloudWatch provides Metric Streams, which forward all metrics captured by Amazon during normal use of AWS offerings to third-party destinations, including Honeycomb.
Metrics stored in AWS CloudWatch can be streamed to other systems using AWS Kinesis Data Firehose.
Honeycomb provides an endpoint that is compatible with CloudWatch Metric Streams, and stores the data it receives in a dataset for easy querying.
For a full list of possible data sources, refer to AWS documentation for services that publish metrics to CloudWatch Metrics.
Honeycomb provides an agentless integration for ingesting S3 logfiles and sending them to Honeycomb as structured events.
AWS services listed in the the table below use a workflow, as shown by the diagram below, to send Amazon S3 logs to Honeycomb.
The integration uses a Lambda function, which is subscribed to
PutObject events on your bucket.
The Lambda source code is available on GitHub.
This is a complete list of AWS services logging to S3 that are supported by Honeycomb AWS Integrations.
|AWS Services supported via S3 Logs||Structured||Unstructured|
|Elastic Load Balancing (ELB) Access Logs||✔|
|Amazon S3 Access Logs||✔|
|Amazon Virtual Private Cloud (VPC)||✔|