This section describes the mechanics of Honeycomb AWS Integrations that collect logs and metrics. It describes various collection methods and documents which AWS services use those methods.
For more details, visit the Honeycomb AWS Integrations GitHub repository for Terraform or CloudFormation.
Many AWS services publish logs to CloudWatch Logs. By default, most AWS services send unstructured logs to CloudWatch. Because Honeycomb is designed to work best with structured events, Honeycomb AWS Integrations convert some of those CloudWatch log streams into structured data.
Deploy the Honeycomb CloudWatch Logs integration with either Terraform or CloudFormation.
AWS CloudWatch provides Subscription Filters, which can be configured to forward all logs in a log group to a Kinesis Firehose destination that can then stream the logs to our Honeycomb Kinesis endpoint. AWS services listed in the unstructured data column of the table below use a workflow, as shown by the diagram below, to send AWS CloudWatch logs to Honeycomb.
Kinesis Data Firehose can invoke Lambda functions to transform incoming source data before delivering the transformed data to its destination. Supported AWS services listed in the structured data column of the table below are either JSON-formatted by default or use an additional step in their workflow, as shown by the diagram above, to structure their log data before sending it to Honeycomb.
Honeycomb AWS Integrations are designed to work with all services that logs to CloudWatch.
For a full list of possible data sources, refer to AWS documentation for services that publish logs to CloudWatch Logs.
This list is a selection of supported AWS services that publish logs through CloudWatch Logs and the type of log data Honeycomb can collect from them.
| AWS Services supported through CloudWatch Logs | Structured | Unstructured |
|---|---|---|
| Amazon API Gateway | ✔ | |
| Amazon Aurora | ✔ | |
| AWS CloudHSM | ✔ | |
| AWS CloudWatch | ✔ | |
| AWS CodeBuild | ✔ | |
| AWS CodeDeploy | ✔ | |
| Amazon Cognito | ✔ | |
| AWS Elastic Beanstalk | ✔ | |
| Amazon Elastic Container Service (ECS) | ✔ | |
| Amazon Elastic Kubernetes Service (EKS) | ✔ | |
| Elastic Load Balancing (ELB) | ✔ | |
| Amazon ElastiCache | ✔ | |
| AWS Fargate | ||
| Amazon Keyspaces | ✔ | |
| AWS Lambda | ✔ | |
| Amazon Managed Streaming for Kafka | ✔ | |
| Amazon MQ | ✔ | |
| AWS Network Firewall | ✔ | |
| AWS OpsWorks | ✔ | |
| Amazon Relational Database Service (RDS) | ✔ | |
| Amazon Route 53 | ✔ | |
| Amazon Simple Notification Service (SNS) | ✔ | |
| Amazon Simple Storage Service (S3) | ✔ | |
| AWS Step Functions | ✔ | |
| Amazon Virtual Private Cloud (VPC) | ✔ |
Each field in a structured log (or event) is queryable with Honeycomb. This makes working with structured AWS log data similar to working with other types of data in Honeycomb.
Unstructured logs are received by Honeycomb as events with a verbose message attribute.
All unstructured log data is contained within this single attribute.
For tips on working with unstructured AWS logs, refer to Work with Your AWS Data.
The CloudWatch Logs integration is a great way to get started and gain insights across a variety of AWS services. If you want to customize the structure of the logs, you might want to write your own data transformation Lambda to be used with the Kinesis Data Firehose. Source code for our RDS data transformations can be viewed in the Agentless Integrations GitHub repository as an example.
Honeycomb AWS Integrations support all AWS services that send metrics to CloudWatch.
AWS CloudWatch provides Metric Streams, which forward all metrics captured by Amazon during normal use of AWS offerings to third-party destinations, including Honeycomb.
Deploy the Honeycomb Cloudwatch Metrics integration with either Terraform or CloudFormation.
Metrics stored in AWS CloudWatch can be streamed to other systems using AWS Kinesis Data Firehose.
Honeycomb provides an endpoint that is compatible with CloudWatch Metric Streams, and stores the data it receives in a dataset for easy querying.
For a full list of possible data sources, refer to AWS documentation for services that publish metrics to CloudWatch Metrics.
Honeycomb provides an agentless integration for ingesting S3 logfiles and sending them to Honeycomb as structured events.
Deploy the Honeycomb S3 Logs integration with either Terraform or CloudFormation.
AWS services listed in the the table below use a workflow, as shown by the diagram below, to send Amazon S3 logs to Honeycomb.
The integration uses a Lambda function, which is subscribed to PutObject events on your bucket.
The Lambda source code is available on GitHub.
This is a complete list of AWS services logging to S3 that are supported by Honeycomb AWS Integrations.
| AWS Services supported via S3 Logs | Structured | Unstructured |
|---|---|---|
| AWS CloudFront | ✔ | |
| AWS CloudTrail | ✔ | |
| Elastic Load Balancing (ELB) Access Logs | ✔ | |
| Amazon S3 Access Logs | ✔ | |
| Amazon Virtual Private Cloud (VPC) | ✔ |
