How AWS Integrations Work | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

How AWS Integrations Work

This section describes the mechanics of Honeycomb AWS Integrations that collect logs and metrics. It describes various collection methods and documents which AWS services use those methods.

Refer to the Honeycomb AWS Integrations GitHub repository for Terraform or CloudFormation for more details.

AWS CloudWatch Logs 

Many AWS services publish logs to CloudWatch Logs. By default, most AWS services send unstructured logs to CloudWatch. Because Honeycomb is designed to work best with structured events, Honeycomb AWS Integrations convert some of those CloudWatch log streams into structured data.

Deploy the Honeycomb CloudWatch Logs integration with either Terraform or CloudFormation.

How AWS CloudWatch Logs Integrations Work 

AWS CloudWatch provides Subscription Filters, which can be configured to forward all logs in a log group to a Kinesis Firehose destination that can then stream the logs to our Honeycomb Kinesis endpoint. AWS services listed in the unstructured data column of the table below use a workflow, as shown by the diagram below, to send AWS CloudWatch logs to Honeycomb.

An architecture diagram depicting AWS CloudWatch log groups pointed at AWS Kinesis Firehose. Firehose is subsequently forwarding data to Honeycomb.

Kinesis Data Firehose can invoke Lambda functions to transform incoming source data before delivering the transformed data to its destination. AWS services listed in the structured data column of the table above are either JSON-formatted by default or use an additional step in their workflow, as shown by the diagram below, to structure their log data before sending it to Honeycomb.

Supported AWS Services Through CloudWatch Logs 

Honeycomb AWS Integrations are designed to work with all services that logs to CloudWatch.

For a full list of possible data sources, refer to AWS documentation for services that publish logs to CloudWatch Logs.

This list is a selection of supported AWS services that publish logs through CloudWatch Logs and the type of log data Honeycomb can collect from them.

AWS Services supported through CloudWatch Logs Structured Unstructured
Amazon API Gateway
Amazon Aurora
AWS CloudHSM
AWS CloudWatch
AWS CodeBuild
AWS CodeDeploy
Amazon Cognito
AWS Elastic Beanstalk
Amazon Elastic Container Service (ECS)
Amazon Elastic Kubernetes Service (EKS)
Elastic Load Balancing (ELB)
Amazon ElastiCache
AWS Fargate
Amazon Keyspaces
AWS Lambda
Amazon Managed Streaming for Kafka
Amazon MQ
AWS Network Firewall
AWS OpsWorks
Amazon Relational Database Service (RDS)
Amazon Route 53
Amazon Simple Notification Service (SNS)
Amazon Simple Storage Service (S3)
AWS Step Functions
Amazon Virtual Private Cloud (VPC)

Working With Unstructured AWS Logs 

Each field in a structured log (or event) is queryable with Honeycomb. This makes working with structured AWS log data similar to working with other types of data in Honeycomb.

Unstructured logs are received by Honeycomb as events with a verbose message attribute. All unstructured log data is contained within this single attribute. For tips on working with unstructured AWS logs, refer to Work with Your AWS Data.

Advanced Use 

The CloudWatch Logs integration is a great way to get started and gain insights across a variety of AWS services. If you want to customize the structure of the logs, you might want to write your own data transformation Lambda to be used with the Kinesis Data Firehose. Source code for our RDS data transformations can be viewed in the Agentless Integrations GitHub repository as an example.

AWS CloudWatch Metrics 

Honeycomb Metrics is available as part of Honeycomb Enterprise and Pro plans.

Honeycomb AWS Integrations support all AWS services that send metrics to CloudWatch.

AWS CloudWatch provides Metric Streams, which forward all metrics captured by Amazon during normal use of AWS offerings to third-party destinations, including Honeycomb.

Deploy the Honeycomb Cloudwatch Metrics integration with either Terraform or CloudFormation.

Note that AWS charges its customers to use this Cloudwatch Metrics API. Please refer to Amazon for specifics on associated egress costs.

How AWS CloudWatch Metrics Integrations work 

Metrics stored in AWS CloudWatch can be streamed to other systems using AWS Kinesis Data Firehose.

Honeycomb provides an endpoint that is compatible with CloudWatch Metric Streams, and stores the data it receives in a dataset for easy querying.

An architecture diagram depicting AWS CloudWatch Metrics pointed at AWS Kinesis Firehose. Firehose is subsequently forwarding data to Honeycomb.

Supported AWS Services Through CloudWatch Metrics 

For a full list of possible data sources, refer to AWS documentation for services that publish metrics to CloudWatch Metrics.

Amazon S3 Logs 

Honeycomb provides an agentless integration for ingesting S3 logfiles and sending them to Honeycomb as structured events.

Deploy the Honeycomb S3 Logs integration with either Terraform or CloudFormation.

How Amazon S3 Logs Integrations Work 

AWS services listed in the the table below use a workflow, as shown by the diagram below, to send Amazon S3 logs to Honeycomb. The integration uses a Lambda function, which is subscribed to PutObject events on your bucket. The Lambda source code is available on GitHub.

An architecture diagram depicting S3 logs pointed at a Lambda which subsequently delivers structured logs to Honeycomb.

Supported AWS Services Through Amazon S3 Logs 

This is a complete list of AWS services logging to S3 that are supported by Honeycomb AWS Integrations.

AWS Services supported via S3 Logs Structured Unstructured
AWS CloudFront
AWS CloudTrail
Elastic Load Balancing (ELB) Access Logs
Amazon S3 Access Logs
Amazon Virtual Private Cloud (VPC)

Did you find what you were looking for?