Skip to main content

​
Overview

If you use OpenTelemetry, AWS manages an official OpenTelemetry Lambda layer called AWS Distro for OpenTelemetry (ADOT) Lambda, which lets you use OpenTelemetry in your Lambda functions and export data asynchronously from those functions.
Using Legacy ADOT Lambda Layers for HoneycombThis guide uses the legacy ADOT Lambda layers that include an embedded OpenTelemetry Collector. AWS now offers newer optimized layers without a collector, but these are designed to export only to AWS CloudWatch and X-Ray.To send telemetry to Honeycomb (a non-AWS endpoint), you need the legacy layers with the embedded collector. AWS acknowledges this use case, stating: “Unless you want to export the telemetry data to a non CloudWatch endpoint, the approach below is not recommended.”
The legacy ADOT Lambda Layer works by embedding a stripped-down version of the OpenTelemetry (OTel) Collector inside an AWS Lambda extension layer. To use it, configure an OTLP exporter to send to the OTel Collector in the Lambda layer. The instructions in this guide clarify and expand on the steps provided in AWS’s documentation for the legacy ADOT Lambda layers with embedded collector.

​
Languages

AWS maintains pre-configured Lambda layers that provide automatic instrumentation and do not require you to modify code. These exist for several languages, including:
  • Java
  • Python
  • JavaScript
Two instrumentation paths exist for ADOT Lambda for Java: the Java SDK or the Java Auto-Instrumentation Agent.Tradeoffs exist between these paths, requiring you to choose either convenience or reduced overhead and cold start latency.Using the Java Auto-Instrumentation Agent simplifies implementation, but adds overhead and increases cold start latency. Using the Java SDK reduces runtime and cold start latency, but requires you to manually instrument and modify your code to get a similar level of observability data.
AWS offers additional Lambda layers that require you to modify your code to add instrumentation. You must import the OpenTelemetry packages directly into your Lambda function and configure them. These layers are available for the following languages:
  • .NET
  • Go
If you are using another language or want to manually build and configure a layer, visit AWS’s documentation on Manual Steps for Private Lambda Layers.

​
Before You Begin

Before you begin, you’ll need:
  • an AWS account with permissions to:
    • Create AWS Lambda functions
    • Access AWS X-Ray
    • Create AWS Identity and Access Management (IAM) policies
  • a Honeycomb Ingest API Key

​
Create a Lambda Function in AWS Lambda

Begin by creating a Lambda function, which is a serverless compute service that lets you run code in response to events without managing servers.
  1. In the AWS Management Console, navigate to the Lambda console, and select Functions from the nav.
  2. Select Create function.
  3. Choose Author from Scratch as the method of creating your function.
  4. Locate the Basic Information section, and enter details for your Lambda function:
    FieldDescription
    Function nameName that describes the purpose of your function.
    RuntimeLanguage to use to write your function.
    ArchitectureInstruction set architecture you want for your function code. We recommend that you choose x86-64.
  5. Locate the Permissions section, and expand it.
  6. For Execution role, select Create a new role with basic Lambda permissions.
  7. Select Create function
  8. In the Lambda code editor window that opens, enter your Lambda function. For example, for JavaScript: 
    exports.lambdaHandler = async (event, context) => {
  response = {
    'statusCode': 200,
    'body': json.stringify('Hello, World!')
}

  return response;
};
  9. Press [Ctrl] + [s] on your keyboard to save your file.
  10. Select Deploy.
    Remember to select Deploy any time you make a change to your Lambda function.

​
Add the AWS Distro for OpenTelemetry Lambda Layer

If you are using .NET or Go, you must instrument your code before you complete this step. Refer to AWS instructions on .NET instrumentation or Go instrumentation.If you are using Java, JavaScript, and Python, your code will be instrumented in a later step.
After you have entered the details and code for your Lambda function, add the ADOT Lambda layer to it.
  1. Locate the Layers section, and select Add a layer.
  2. In the Add layer window, locate Layer source, and select Specify an ARN.
  3. Locate the Specify an ARN section, and copy and paste the example ARN from the AWS legacy layer documentation for your language:
  4. Modify your ARN:
    • Replace the <architecture> placeholder in your ARN with the appropriate value. For x86-based processors, use amd64.
    • Replace the <region> placeholder in your ARN with the appropriate value from the supported region values listed in the AWS documentation linked in the previous step. Lambda layers can be used only in the region in which they are published, so make sure to use the layer in the same region as your Lambda function.
  5. Select Add.

​
Disable Active Tracing

We recommend disabling the Active Tracing setting for AWS X-Ray because:
  • If you are visualizing your data with Honeycomb, then exporting data to AWS X-Ray is unnecessary.
  • Active Tracing adds extra spans, which cannot be exported outside of AWS, to traces by default.
  • Disabling Active Tracing provides some data savings and avoids conflicts with other tracing instrumentation you may have.
To disable Active Tracing:
  1. In the Configuration tab, select Monitoring and operations tools, and then select Edit.
  2. Locate the AWS X-Ray section, and toggle off Active tracing.
  3. Select Save.

​
Configure OpenTelemetry Packages

If you are using .NET or Go, skip this step and proceed to Customize the ADOT Collector to Send Telemetry Data to Honeycomb.
Import the OpenTelemetry packages directly into your Lambda function and configure them using the reserved AWS_LAMBDA_EXEC_WRAPPER environment variable. The environment variable points to a script that performs the configuration directly in each layer, so you can avoid modifying code.
  1. In the Configuration tab of the Lambda console, select Environment variables, and then select Edit.
  2. For Key, enter AWS_LAMBDA_EXEC_WRAPPER. AWS_LAMBDA_EXEC_WRAPPER is a reserved environment variable AWS Lambda uses to identify a custom wrapper script to run before your function’s main handler.
  3. For Value, enter /opt/otel-handler. This script will invoke your Lambda application with the automatic instrumentation applied. For Java, you can instrument using either the Java SDK or the Java Auto-Instrumentation Agent. When using the Java SDK, scripts for several other handler types exist.
  4. Select Add environment variable.

​
Customize the ADOT Collector to Send Telemetry Data to Honeycomb

To send your telemetry data to Honeycomb, you must customize the embedded ADOT Collector configuration in the legacy Lambda layer. The ADOT Lambda Layers support a variety of confmap providers, which are types of OpenTelemetry Collector components responsible for fetching configuration from a URI. In this example, we use a file confmap provider.
  1. In the Code tab of the Lambda console, add a new file named otel-collector-config.yaml alongside your Lambda function (index.js).
  2. Enter the following code, which configures the Collector to send traces to Honeycomb: 
    
#otel-collector-config.yaml in the root directory
#Set an environment variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/otel-collector-config.yaml'

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp:
    endpoint: "api.honeycomb.io:443" # US instance
    #endpoint: "api.eu1.honeycomb.io:443" # EU instance
    headers:
      "x-honeycomb-team": "YOUR_HONEYCOMB_INGEST_API_KEY"
      #"x-honeycomb-dataset": "YOUR_DATASET" #If you are a Honeycomb Classic user, you must also specify the Dataset for traces. 

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]
    Remember to replace the YOUR_HONEYCOMB_INGEST_API_KEY placeholder with your Honeycomb Ingest API Key.
    To explore additional Collector configurations customized for Honeycomb, visit Send Data with the OpenTelemetry Collector.
    This configuration file uses batch processing, which is our recommended processing method. Ideally, you would also apply filtering in this file. Alternatively, you could send single events to a second gateway Collector and have it perform filtering and batching.
  3. Press [Ctrl] + [s] on your keyboard to save your file.
  4. Select Deploy.
    Remember to select Deploy any time you make a change to your Collector configuration file.

​
Configure the ADOT Collector to Find Your Custom Configuration

After you have customized your Collector configuration file, you must tell the embedded ADOT Collector where to find your custom configuration using the reserved OPENTELEMETRY_COLLECTOR_CONFIG_FILE environment variable.
  1. In the Configuration tab, select Environment variables, and then select Edit.
  2. For Key, enter OPENTELEMETRY_COLLECTOR_CONFIG_FILE. OPENTELEMETRY_COLLECTOR_CONFIG_FILE is a reserved environment variable AWS Lambda uses to identify a custom configuration file for the OpenTelemetry Collector.
  3. For Value, enter /var/task/otel-collector-config.yaml.
    You add the /var/task/ prefix to your file path because you used a file configmap provider.
  4. Select Add environment variable.

​
Instrument Your Lambda Function

If you are using .NET or Go, skip this step. You should have already added instrumentation to your code (before adding the AWS Distro for OpenTelemetry Lambda Layer).
Languages with automatic instrumentation, which lets data flow automatically into Honeycomb, include:
  • Java
  • Python
  • JavaScript
Automatic instrumentation works only if you use an automatic instrumentation library. In this example, we use https.get.
  1. Import the https library and configure it to use Honeycomb. For this JavaScript example, add the following code in a location above the exports.lambdaHandler line in the Lambda function you created in the first step of this guide: 
    import https from `https`;
var options = {
    hostname: 'honeycomb.io'
    , path: '/'
    , method: 'GET'
};
  2. Instruct the library to return data. Add the following code in a location directly above the return response line in the Lambda function you created in the first step of this guide: 
    var req = https.request(
  options, function(res) {
    res.on('data', function(d) {
      //
    });
  }
);
req.end();
    The full code should look similar to: 
    import https from `https`;
var options = {
    hostname: 'honeycomb.io'
    , path: '/'
    , method: 'GET'
};

exports.lambdaHandler = async (event, context) => {
  response = {
    'statusCode': 200,
    'body': json.stringify('Hello, World!')
  }

  var req = https.request(
    options, function(res) {
      res.on('data', function(d) {
        //
      });
    }
  );  
  req.end();

  return response;
};
  3. Press [Ctrl] + [s] on your keyboard to save.
  4. Select Deploy.

​
Test Your Implementation

It’s time to test your implementation and see data in Honeycomb!
  1. In the Code tab, select Test.
  2. Open Honeycomb and look for a Dataset with the same name as the Lambda function you created in the first step of this guide.
If you see the Dataset, then your data is flowing to Honeycomb!
To learn how to explore your AWS data, visit Investigate AWS Data in Honeycomb.

​
Creating a Function URL

To test your Lambda function more easily, you can use a function URL, which is a dedicated HTTP(S) endpoint that you can enable for your Lambda function. You can use the function URL to automate testing. To create a function URL:
  1. In the Lambda console, select Functions from the nav.
  2. Locate and select the name of the function for which you you want to create a function URL.
  3. Select the Configuration tab, and then select Function URL.
  4. Select Create function URL.
  5. For Auth type, choose AWS_IAM or NONE, depending on who you would like to have access to your function URL.
  6. Select Save.

​
Propagating Trace Context

Trace context plays a critical role in linking together individual services into a cohesive trace. In AWS Lambda, ensuring that trace context is correctly passed across multiple Lambda functions is essential for accurate, end-to-end tracing.

​
Disabling AWS Context Propagation

By default, AWS Lambda automatically propagates trace context between services. If you need to disable automatic context propagation, you can use either an environment variable or a configuration file.

​
Disabling Context Propagation Using an Environment Variable

Set the OTEL_LAMBDA_DISABLE_CONTEXT_PROPAGATION reserved environment variable to true: 
OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true

​
Disabling Context Propagation Using a Configuration File

Using a configuration file gives you more control over how trace context is managed in your Lambda functions. To disable context propagation using a configuration file:
  1. Create a configuration file (in this example, named lambda-config.js), and add the following code: 
    // lambda-config.js
global.configureLambdaInstrumentation = (config) => {
  return {
    ...config,
    disableAwsContextPropagation: true
  }
}
  2. Include this configuration file when starting your Lambda function, either as part of the start command or when using the NODE_OPTIONS reserved environment variable: 
    NODE_OPTIONS=--require ./lambda-config.js

​
Propagating Context Using Lambda Event Arguments

When invoking a Lambda function outside of an HTTP context, the trace context may not be included automatically. In these cases, you can use Lambda event arguments to propagate the context manually. For example, in Node.js: 
const { trace, context, TraceFlags } = require('@opentelemetry/api');

let tracer = trace.getTracer("aws-lambda-tracer");

exports.handler = async function (event, ctx) {
  let newContext = trace.setSpanContext(context.active(), {
    traceId: event.traceId,
    spanId: event.spanId,
    traceFlags: TraceFlags.SAMPLED,
    isRemote: true });

  context.with(newContext, () => {
    const span = tracer.startSpan('handler function', newContext);
    span.end();
  });

  return context.logStreamName;
};

​
Troubleshooting

If you need to troubleshoot, explore these solutions to common issues.

​
Missing root spans in traces

If you are using AWS Lambda with API Gateway (or another service that governs traffic), missing root spans for your traces in Honeycomb are likely. This is because API Gateway often generates the initial request but may not propagate tracing headers correctly to your Lambda function. Some solutions you can try to ensure your traces include root spans include:
  • Disable Active Tracing on your Lambda function (in the AWS Console under Configuration > Monitoring).
    If you are using Terraform to launch your Lambda function, you can turn off tracing by assigning the Passthrough value to the tracing_config argument. Learn more about the tracing_config argument in Terraform’s documentation for Lambda Function resources.
  • To ensure consistent trace context, add the OTEL_PROPAGATORS reserved environment variable to your Lambda function (in the AWS Console under Configuration > Environment variables) and set its value to tracecontext.
  • If you call your Lambda function directly from code, make sure the traceparent header is present. With the traceparent header, OpenTelemetry can recognize the request as part of a larger trace and will create a root span. If the traceparent header is missing, add a code block to wrap your entrypoint function in a span. For example, in JavaScript: 
    // index.js
const { trace, context, SpanStatusCode, propagation } = require('@opentelemetry/api');

let tracer = trace.getTracer('tracer.name.here');

exports.handler = async (event) => {
  let span = null;
  if (!event.headers.traceparent) {
    span = tracer.startActiveSpan('something.different', { root: true });
  }

  const response = {
    statusCode: 200,
    body: createTrackingId(event.body || 'salt'),
  };

  if (span) {
    span.end();
  }

  return response;
};
  • Ensure your OpenTelemetry Collector is correctly configured to send data to Honeycomb.
  • Make sure you have selected an appropriate service.name. By default, the AWS Lambda layer sets service.name to the Lambda function’s name. Because Honeycomb creates a dataset for each service.name, the default AWS Lambda behavior can cause traces from related Lambda functions to be split across different datasets. To keep datasets together, override the default service.name by explicitly setting the OTEL_SERVICE_NAME reserved environment variable for each Lambda function.

​
Missing non-root spans in traces

Although the AWS Lambda Setup documentation includes a step that enables Active Tracing, we recommend disabling Active Tracing. When Active Tracing is enabled, Lambda uses a ParentBased sampler with a sample rate of 5%. The AWS Propagator code always creates a new context, which can cause some spans to be dropped when using a ParentBased sampler. With this sampler, if spans are dropped and DEBUG mode is enabled, you might see the following error in the logs: 
DEBUG Recording is off, propagating context in a non-recording span
To resolve this, use an AlwaysOn sampler by setting the OTEL_TRACES_SAMPLER reserved environment variable to always_on: 
OTEL_TRACES_SAMPLER=always_on
To confirm the sampler configuration in use, output details to the console: 
let tracer = trace.getTracer('<TRACER_NAME_HERE>');
console.log(`Tracer sampler information: ${tracer['_sampler'].toString()}`)

​
Lambda layer not instrumenting code

Ensure that you have created the AWS_LAMBDA_EXEC_WRAPPER reserved environment variable, which is essential for initializing OpenTelemetry instrumentation, and have set it to use the OpenTelemetry handler: 
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler