Getting AWS Cloudwatch Logs into Honeycomb | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Getting AWS Cloudwatch Logs into Honeycomb

Honeycomb provides an agentless integration for ingesting CloudWatch Logs. The integration runs as one or more Lambda functions, subscribed to one or more of your CloudWatch Log Group(s).

The source is available on Github and instructions for getting started are provided here. Do you have a use case not covered here? Please open an issue.

Prerequisites 

You will need permission to deploy a Cloudformation stack with an IAM role in your AWS account.

Install 

To install, use one of the AWS Cloudformation Quick-Create links below. These links will launch the AWS Cloudformation console with the appropriate template and steer you through the installation process.

Cloudformation Stack Creation

Generic JSON Integration 

This integration accepts lines with arbitrary JSON. If you are already writing structured logs in JSON format, this is what you want! Click Here

You will need to provide the following parameters:

  • Stack Name
  • Cloudwatch Logs Group Name (you can supply up to 6 per installation)
  • Your Honeycomb API Key (optionally encrypted)
  • Honeycomb Dataset Name

Optionally, you can supply:

  • Sample rate
  • The ID of the AWS Key Management Service key used to encrypt your API Key. If your API Key is not encrypted, do not set a value here

Example Log Format 

The integration expects each line to contain a JSON object and nothing else.

{"field1": "data1", "field2": "data2", "field3": 12345, "field4": {"field5": false}}
{"field1": "data1", "field2": "data2", "field3": 12345, "field4": {"field5": false}}

Regex Integration 

If your logs are not structured with JSON but you can write an (re2) regex to parse them, use this integration. Click Here

  • Stack Name
  • Cloudwatch Logs Group Name (you can supply up to 6 per installation)
  • Your Honeycomb API Key (optionally encrypted)
  • Honeycomb Dataset Name
  • re2 compatible regex pattern

Honeycomb columns are generated by defining named capture groups. For example, (?P<name>re) would create a column called “name” if successfully parsed. Here are some example regexes for specific log formats:

AWS ELB Logs 

(?P<timestamp>.+) (?P<elb>.+) (?P<client_authority>.+) (?P<backend_authority>.+) (?P<request_processing_time>.+) (?P<backend_processing_time>.+) (?P<response_processing_time>.+) (?P<elb_status_code>.+) (?P<backend_status_code>.+) (?P<received_bytes>.+) (?P<sent_bytes>.+) (?P<request>".+") (?P<user_agent>".+") (?P<ssl_cipher>.+) (?P<ssl_protocol>.+)

AWS VPC Flow Logs 

(?P<version>\d+) (?P<account_id>\d+) (?P<interface_id>eni-[0-9a-f]+) (?P<src_addr>[\d\.]+) (?P<dst_addr>[\d\.]+) (?P<src_port>\d+) (?P<dst_port>\d+) (?P<protocol>\d+) (?P<packets>\d+) (?P<bytes>\d+) (?P<start_time>\d+) (?P<end_time>\d+) (?P<action>[A-Z]+) (?P<log_status>[A-Z]+)

Optionally, you can supply:

  • Sample rate
  • The ID of the AWS Key Management Service key used to encrypt your API Key. If your API Key is not encrypted, do not set a value here.

VPC Flow Logs 

If you have enabled VPC Flow Logs, you can get quick insight into your AWS network with the VPC Flow Log Integration. VPC Flow Logs go to Cloudwatch Logs, so ingesting them is a matter of installing our integration and pointing it at the correct log group. Click Here

You will need to provide the following parameters:

  • Stack Name
  • Cloudwatch Log Group Name that receives your VPC Flow Logs
  • Your Honeycomb API Key (optionally encrypted)
  • Honeycomb Dataset Name

Optionally, you can supply:

  • Sample rate (recommended if you are ingesting events from a busy production network)
  • The ID of the AWS Key Management Service key used to encrypt your API Key. If your API Key is not encrypted, do not set a value here

Encrypting Your API Key 

When installing the integration, you must supply your Honeycomb API Key via Cloudformation parameter. Cloudformation parameters are not encrypted, and are plainly viewable to anyone with access to your Cloudformation stacks or Lambda functions. For this reason, we strongly recommend that your Honeycomb API Key be encrypted. To encrypt your key, use AWS’s KMS service.

First, you will need to create a KMS key if you do not have one already. The default account keys are not suitable for this use case.

$ aws kms create-key --description "used to encrypt secrets"
{
    "KeyMetadata": {
        "AWSAccountId": "123455678910",
        "KeyId": "a38f80cc-19b5-486a-a163-a4502b7a52cc",
        "Arn": "arn:aws:kms:us-east-1:123455678910:key/a38f80cc-19b5-486a-a163-a4502b7a52cc",
        "CreationDate": 1524160520.097,
        "Enabled": true,
        "Description": "used to encrypt honeycomb secrets",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER"
    }
}
# optionally, create an alias for the KMS key to describe the key's usage: 
$ aws kms create-alias --alias-name alias/secrets_key --target-key-id=a38f80cc-19b5-486a-a163-a4502b7a52cc

Save a file containing only your Honeycomb API Key to be passed into the encryption step. For example, if abc123 is your Honeycomb API Key and my-key is the name of the file, create the file like this:

echo -n abc123 > my-key

Next, encrypt your Honeycomb API Key:

$ aws kms encrypt --key-id=a38f80cc-19b5-486a-a163-a4502b7a52cc --plaintext fileb://my-key
{
    "CiphertextBlob": "AQICAHge4+BhZ1sURk1UGUjTZxmcegPXyRqG8NCK8/schk381gGToGRb8n3PCjITQPDKjxuJAAAAcjBwBgkqhkiG9w0BBwagYzBhAgEAMFwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM0GLK36ChLOlHQiiiAgEQgC9lYlR3qvsQEhgILHhT0eD4atgdB7UAMW6TIAJw9vYsPpnbHhqhO7V8/mEa9Iej+g==",
    "KeyId": "arn:aws:kms:us-east-1:702835727665:key/a38f80cc-19b5-486a-a163-a4502b7a52cc"
}

Record the CiphertextBlob and the last part of the KeyId from the encryption step. In the example above, the last part of the KeyId is a38f80cc-19b5-486a-a163-a4502b7a52cc. Enter the CiphertextBlob into the Cloudformation template as the HoneycombWriteKey. Enter the KeyId into the Cloudformation template as the KMSKeyId.

For more information about the need to use fileb:// prefix, see the AWS Reference Guide.

Troubleshooting 

Integration Logs 

The Cloudwatch Logs integration is just a normal Lambda function, which means you can see its metrics and log messages from the Lambda Console. Look for functions starting with CloudwatchLambdaHandler. From there, you can view error rate, latency, and Cloudwatch logs.

If you do not see events in Honeycomb, there may be errors returned from the Honeycomb API. To see API responses and enable debugging, add the HONEYCOMB_DEBUG=true environment variable to the Lambda function.

Updating/Redeploying 

If you are trying to pick up a newer version of the integration, or have misconfigured an existing installation, it is better to completely delete the CFN stack and re-create it using the quick-create links.

Advanced Use 

Quick-create links are great for getting started, but if you have an existing workflow for configuring infrastructure, you might want to directly configure the Lambda functions yourself to suit your needs. We have provided example templates for Cloudformation and Terraform in our repository to get you started.

Did you find what you were looking for?