AWS Lambda Instrumentation | Honeycomb

Honeycomb.io Documentation
  1. Docs
  2. Integrations
  3. Amazon Web Services (AWS)
  4. AWS Lambda

AWS Lambda Instrumentation

For serverless apps based on Lambda and similar platforms, observability can be challenging. With no server to log into, clunky logging interfaces, and short, event-driven process runtimes, making sense of what is going on is often difficult.

To help with this, an official AWS-managed OpenTelemetry Lambda Layer lets you use OpenTelemetry in your Lambda Functions and export data asynchronously from those functions.

If you are not using OpenTelemetry, Honeycomb also has an extension that can run as a Lambda Layer, or inside a container alongside your lambda function.

AWS Managed OpenTelemetry Lambda Layer 

The AWS-managed OpenTelemetry Lambda Layer works by embedding a stripped-down version of the OpenTelemetry (OTel) Collector inside an AWS Lambda Extension Layer. To use it in any language, configure an OTLP exporter to send to the OTel Collector in the Lambda Layer.

AWS maintains pre-configured Lambda Layers for several languages with automatic instrumentation:

  • Java
  • Python
  • Nodejs
  • .NET
  • Go

See AWS’s documentation on Getting Started with AWS Lambda Layers for more information.

If not using any of the above listed languages, or would prefer to manually build and configure a layer, refer to AWS’s documentation on Manual Steps for Private Lambda Layers.

Fix missing root spans in traces 

If using Lambda with API Gateway or another service that governs traffic, missing root spans for your traces in Honeycomb are likely. To mitigate, complete these steps:

  1. Disable AWS X-Ray tracing on your Lambda in the AWS UI (Configuration > Monitoring).

    If using Terraform to launch your lambda function, you can turn off tracing with the tracing_config option PassThrough. Learn more here.

  2. Add an environment variable named OTEL_PROPAGATORS with value tracecontext to your Lambda in the AWS UI (Configuration > Environment variables).

  3. Optional: If you call your Lambda directly from code and the traceparent header does not exist, add a code block to wrap your initial entrypoint function with a span. Otherwise, skip this step. For example, if using JavaScript:

// index.js
const { trace, context, SpanStatusCode, propagation } = require('@opentelemetry/api');

let tracer = trace.getTracer('tracer.name.here');

exports.handler = async (event) => {
  let span = null;
  if (!event.headers.traceparent) {
    span = tracer.startActiveSpan('something.different', { root: true });
  }

  const response = {
    statusCode: 200,
    body: createTrackingId(event.body || 'salt'),
  };

  if (span) {
    span.end();
  }

  return response;
};
  1. Ensure your OpenTelemetry Collector configuration is correctly configured to send to Honeycomb.
  2. Ensure a good service.name for your Lambdas. By default, the AWS Lambda Layer uses the Lambda name for the service.name resource attribute. If you want to keep datasets together for related lambdas, this default behavior can cause problems. Override the default service name by defining the OTEL_SERVICE_NAME environment variable for each Lambda.

After following the above steps, traces in Honeycomb should no longer have missing root spans.

Fix missing other spans in traces 

AWS Lambda Setup docs include a step to enable Active Tracing for automatic instrumentation. Our step above to fix missing root spans suggest disabling Active Tracing. When Active tracing is enabled in a Lambda function, it defaults to a ParentBased Sampler, with a sample rate of 5%. The AWS Propagator code always creates a new context, thus resulting in some spans not being recorded when using a ParentBased Sampler.

When spans are dropped with this sampler and DEBUG is enabled, the following error may be logged:

DEBUG Recording is off, propagating context in a non-recording span

To solve this, set the sampler to an AlwaysOn sampler using the OTEL_TRACES_SAMPLER environment variable:

OTEL_TRACES_SAMPLER=always_on

One way to confirm the sampler configuration in use is to output the details to the console:

let tracer = trace.getTracer('tracer.name.here');
console.log(`Tracer sampler information: ${tracer['_sampler'].toString()}`)

Honeycomb Lambda Extension 

If not using OpenTelemetry, you can use the Honeycomb Lambda Extension. This extension is designed to run alongside your Lambda Function. It integrates with the Lambda Logs API and receives log messages from your lambda function, which are then sent to Honeycomb as events. Structured log messages sent to stdout or stderr from your lambda function will be sent to Honeycomb as events.

The extension can be run inside a container or added as a Lambda Layer.

Overview of Lambda Extension

How It Works 

AWS Lambda Extensions allow you to extend the functionality of Lambda functions through configuration as Lambda Layers or run inside a container image.

Extensions run alongside the Lambda Runtime and can read data environment variables from the environment.

Once the Honeycomb Lambda Extension is configured, any structured logs emitted to stdout or stderr by your Lambda function will be parsed by the extension and sent to a dataset that you specify in Honeycomb.

Note
Logs emitted in a Lambda invocation will be parsed during the following invocation, and any remaining logs will be parsed whenever the Lambda runtime shuts down. This means that you may experience some delay in seeing events arrive in Honeycomb, especially if your Lambda function receives low traffic.

Architectures 

The Honeycomb Lambda Extension is available as an external extension pre-built in a lambda layer for either x86_64 or arm64 architectures.

Note
Graviton2 arm64 is supported in most, but not all regions. See AWS Lambda Pricing for which regions are supported.

Installing as a Lambda Layer 

To start using the Honeycomb Lambda Extension, add the extension to your function as a Lambda Layer.

The extension can be added as a Lambda Layer with the AWS CLI tool: 
aws lambda update-function-configuration --function-name YourFunctionName \
    --layers "arn:aws:lambda:us-east-1:702835727665:layer:honeycomb-lambda-extension-x86_64-v11-1-1:1"

You can also use infrastructure-as-code tools such as Terraform:

resource "aws_lambda_function" "extensions-demo-example-lambda" {
  function_name = "HoneycombExtensionExample"
  ...
  layers = ["arn:aws:lambda:us-east-1:702835727665:layer:honeycomb-lambda-extension-x86_64-v11-1-1:1"]
}

Add the following environment variables to your Lambda function configuration:

  • LIBHONEY_DATASET - The Honeycomb dataset you would like events to be sent to. This could be the service name representing the function or a generic dataset name representing the data.
  • LIBHONEY_API_KEY - Your Honeycomb API Key (also called Write Key).

As well as the following optional environment variable:

  • LIBHONEY_API_HOST - Mostly used for testing purposes, or to be compatible with proxies. Defaults to https://api.honeycomb.io/.
  • LOGS_API_DISABLE_PLATFORM_MSGS - Optional. Set to “true” in order to disable “platform” messages from the logs API.
  • HONEYCOMB_DEBUG - Optional. Set to “true” to enable debug statements and troubleshoot issues. Enabling this will subscribe to Libhoney’s response queue and log the success or failure of sending events to Honeycomb.
  • HONEYCOMB_BATCH_SEND_TIMEOUT - Optional. Default: “15s” (15 seconds; refer to note below about timeout durations). The timeout for the complete HTTP request/response cycle for sending a batch of events Honeycomb. A batch send that times out has a single built-in retry; total time a lambda invocation may spend waiting is double this value. A very low duration may result in duplicate events, if Honeycomb data ingest is successful but slower than this timeout (rare, but possible).
  • HONEYCOMB_CONNECT_TIMEOUT - Optional. Default: 3s (3 seconds; refer to note below about timeout durations). The timeout for establishing a TCP connection to Honeycomb. This is useful when there are connectivity issues between your Lambda environment and Honeycomb, allowing upload requests to fail faster and avoid waiting for the longer batch send timeout to elapse.
Note
TIMEOUT options should be given a value in a format parseable as a duration, such as “1m”, “15s”, or “750ms”. There are other valid time units (“ns”, “us”/“µs”, “h”), but their use does not fit a timeout for HTTP connections made in the AWS Lambda compute environment.

Installing in a Container Image 

AWS Lambda functions can now be packaged and deployed as container images. This allows developers to leverage the flexibility and familiarity of container tooling, workflows and dependencies.

To run the Honeycomb Lambda Extension inside your container image, you must download the extension and place it in the /opt/extensions directory of your image.

FROM amazon/aws-lambda-ruby:2.7

ARG FUNCTION_DIR="/var/task"

RUN mkdir -p /opt/extensions
RUN yum install -y curl
RUN curl -L https://honeycomb.io/download/honeycomb-lambda-extension/v11.1.0/honeycomb-lambda-extension-<ARCH> -o /opt/extensions/honeycomb-lambda-extension
RUN chmod +x /opt/extensions/honeycomb-lambda-extension

RUN mkdir -p ${FUNCTION_DIR}
COPY . ${FUNCTION_DIR}
ENV LIBHONEY_API_KEY <HONEYCOMB_API_KEY>
ENV LIBHONEY_DATASET <HONEYCOMB_DATASET>

CMD ["app.handler"]

Replace <ARCH> with the appropriate architecture (x86_64 or arm64) based on your container image.

AWS Lambda provides a number of base images you can use. You can also use a custom base image, and adjust the examples accordingly.

Performance Implications 

The Honeycomb Lambda Extension runs independently of your Lambda Function and does not add to your Lambda Function’s execution time or add any additional latency. However, it does increase your Lambda Function’s size in proportion with the size of the Honeycomb Lambda Extension binary.

Examples 

Below are some structured logging examples using some libraries we are familiar with at Honeycomb. Feel free to use your own!

Go  

import (
  log "github.com/sirupsen/logrus"
)

log.SetFormatter(&log.JSONFormatter{})

func Handler(ctx context.Context) error {
  // Measure execution time
  startTime := time.Now()

  // ...

  // Get the Lambda context object
  lc, _ := lambdacontext.FromContext(ctx)

  log.WithFields(log.Fields{
    "function_name": lambdacontext.FunctionName,
    "function_version": lambdacontext.FunctionVersion,
    "request_id": lc.AwsRequestID,
    "duration_ms": time.Since(startTime).Milliseconds(),
    // other fields of interest
  }).Info("Hello World from Lambda!")
}

Python  

import structlog
structlog.configure(processors=[structlog.processors.JSONRenderer()])
log = structlog.get_logger()

def handler(event, context):
    # measure execution time
    start_time = datetime.datetime.now()

    # ...

    log.msg(
        "Hello World from Lambda!",
        function_name=context.function_name,
        function_version=context.function_version,
        request_id=context.aws_request_id,
        duration_ms=(datetime.datetime.now() - start_time).total_seconds() * 1000,
        # other fields of interest
    )

JavaScript  

var bunyan = require('bunyan');
var log = bunyan.createLogger();


module.exports.handler = (event, context, callback) => {
  // Measure execution time
  let startTime = Date.now();

  // ...

  log.info({
    functionName: context.functionName,
    functionVersion: context.functionVersion,
    requestId: context.awsRequestId,
    // Example fields - send anything that seems relevant!
    userId: event.UserId,
    userAction: event.UserAction,
    latencyMs: Date.now() - startTime,
  },
  'Hello World from Lambda!');
}
Note
Do not use console.log to write structured log lines in Lambda. Lambda uses a patched version of console.log, injecting extra information with each line that does not work correctly with the Honeycomb Extension Lambda Logs integration.
On this page: