1. Docs
2. Integrations
3. Amazon Web Services (AWS)
4. AWS Lambda
5. OpenTelemetry

Export Data with AWS Lambda Layer + OpenTelemetry

Table of Contents

Overview 

If you use OpenTelemetry, AWS manages an official OpenTelemetry Lambda layer called AWS Distro for OpenTelemetry (ADOT) Lambda, which lets you use OpenTelemetry in your Lambda functions and export data asynchronously from those functions.

ADOT Lambda Layer works by embedding a stripped-down version of the OpenTelemetry (OTel) Collector inside an AWS Lambda extension layer. To use it, configure an OTLP exporter to send to the OTel Collector in the Lambda layer.

The instructions in this guide clarify and expand on the steps provided in AWS’s Getting Started with AWS Lambda Layers.

Languages 

AWS maintains pre-configured Lambda layers that provide automatic instrumentation and do not require you to modify code. These exist for several languages, including:

• Java
• Python
• JavaScript

AWS offers additional Lambda layers that require you to modify your code to add instrumentation. You must import the OpenTelemetry packages directly into your Lambda function and configure them. These layers are available for the following languages:

• .NET
• Go

If you are using another language or want to manually build and configure a layer, visit AWS’s documentation on Manual Steps for Private Lambda Layers.

Before You Begin 

Before you begin, you’ll need:

• an AWS account with permissions to:
  • Create AWS Lambda functions
  • Access AWS X-Ray
  • Create AWS Identity and Access Management (IAM) policies
• a Honeycomb Ingest API Key

Create a Lambda Function in AWS Lambda 

Begin by creating a Lambda function, which is a serverless compute service that lets you run code in response to events without managing servers.

1. In the AWS Management Console, navigate to the Lambda console, and select Functions from the nav.

2. Select Create function.

3. Choose Author from Scratch as the method of creating your function.

4. Locate the Basic Information section, and enter details for your Lambda function:

   Field	Description
   Function name	Name that describes the purpose of your function.
   Runtime	Language to use to write your function.
   Architecture	Instruction set architecture you want for your function code. We recommend that you choose x86-64.

5. Locate the Permissions section, and expand it.

6. For Execution role, select Create a new role with basic Lambda permissions.

7. Select Create function

8. In the Lambda code editor window that opens, enter your Lambda function. For example, for JavaScript:

   exports.lambdaHandler = async (event, context) => {
     response = {
       'statusCode': 200,
       'body': json.stringify('Hello, World!')
   }
   
     return response;
   };
   

9. Press [Ctrl] + [s] on your keyboard to save your file.

10. Select Deploy.

    Tip

    Remember to select Deploy any time you make a change to your Lambda function.

Add the AWS Distro for OpenTelemetry Lambda Layer 

After you have entered the details and code for your Lambda function, add the ADOT Lambda layer to it.

1. Locate the Layers section, and select Add a layer.

2. In the Add layer window, locate Layer source, and select Specify an ARN.

3. Locate the Specify an ARN section, and copy and paste the example ARN from the AWS documentation for your language:

   • Java SDK
   • Java Auto-instrumentation Agent
   • JavaScript
   • Python
   • .NET
   • Go

4. Modify your ARN:

   • Replace the <architecture> placeholder in your ARN with the appropriate value. For x86-based processors, use amd64.
   • Replace the <region> placeholder in your ARN with the appropriate value from the supported region values listed in the AWS documentation linked in the previous step. Lambda layers can be used only in the region in which they are published, so make sure to use the layer in the same region as your Lambda function.

5. Select Add.

Disable Active Tracing 

We recommend disabling the Active Tracing setting for AWS X-Ray because:

• If you are visualizing your data with Honeycomb, then exporting data to AWS X-Ray is unnecessary.
• Active Tracing adds extra spans, which cannot be exported outside of AWS, to traces by default.
• Disabling Active Tracing provides some data savings and avoids conflicts with other tracing instrumentation you may have.

To disable Active Tracing:

1. In the Configuration tab, select Monitoring and operations tools, and then select Edit.

2. Locate the AWS X-Ray section, and toggle off Active tracing.

3. Select Save.

Configure OpenTelemetry Packages 

Import the OpenTelemetry packages directly into your Lambda function and configure them using the reserved AWS_LAMBDA_EXEC_WRAPPER environment variable. The environment variable points to a script that performs the configuration directly in each layer, so you can avoid modifying code.

Customize the ADOT Collector to Send Telemetry Data to Honeycomb 

To send your telemetry data to Honeycomb, you must customize your ADOT Collector configuration. The ADOT Lambda Layers support a variety of confmap providers, which are types of OpenTelemetry Collector components responsible for fetching configuration from a URI. In this example, we use a file confmap provider.

1. In the Code tab of the Lambda console, add a new file named otel-collector-config.yaml alongside your Lambda function (index.js).

2. Enter the following code, which configures the Collector to send traces to Honeycomb:

   
   #otel-collector-config.yaml in the root directory
   #Set an environment variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/otel-collector-config.yaml'
   
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   
   processors:
     batch:
   
   exporters:
     otlp:
       endpoint: "api.honeycomb.io:443" # US instance
       #endpoint: "api.eu1.honeycomb.io:443" # EU instance
       headers:
         "x-honeycomb-team": "YOUR_HONEYCOMB_INGEST_API_KEY"
         #"x-honeycomb-dataset": "YOUR_DATASET" #If you are a Honeycomb Classic user, you must also specify the Dataset for traces. 
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlp]
   

   Important

   Remember to replace the YOUR_HONEYCOMB_INGEST_API_KEY placeholder with your Honeycomb Ingest API Key.

   To explore additional Collector configurations customized for Honeycomb, visit Send Data with the OpenTelemetry Collector.

   Tip

   This configuration file uses batch processing, which is our recommended processing method. Ideally, you would also apply filtering in this file. Alternatively, you could send single events to a second gateway Collector and have it perform filtering and batching.

3. Press [Ctrl] + [s] on your keyboard to save your file.

4. Select Deploy.

   Tip

   Remember to select Deploy any time you make a change to your Collector configuration file.

Configure the ADOT Collector to Find Your Custom Configuration 

After you have customized your Collector, you must tell the ADOT Collector where to find your custom configuration using the reserved OPENTELEMETRY_COLLECTOR_CONFIG_FILE environment variable.

1. In the Configuration tab, select Environment variables, and then select Edit.

2. For Key, enter OPENTELEMETRY_COLLECTOR_CONFIG_FILE. OPENTELEMETRY_COLLECTOR_CONFIG_FILE is a reserved environment variable AWS Lambda uses to identify a custom configuration file for the OpenTelemetry Collector.

3. For Value, enter /var/task/otel-collector-config.yaml.

   Tip

   You add the /var/task/ prefix to your file path because you used a file configmap provider.

4. Select Add environment variable.

Instrument Your Lambda Function 

Languages with automatic instrumentation, which lets data flow automatically into Honeycomb, include:

• Java
• Python
• JavaScript

Automatic instrumentation works only if you use an automatic instrumentation library. In this example, we use https.get.

1. Import the https library and configure it to use Honeycomb. For this JavaScript example, add the following code in a location above the exports.lambdaHandler line in the Lambda function you created in the first step of this guide:

   import https from `https`;
   var options = {
       hostname: 'honeycomb.io'
       , path: '/'
       , method: 'GET'
   };
   

2. Instruct the library to return data. Add the following code in a location directly above the return response line in the Lambda function you created in the first step of this guide:

   var req = https.request(
     options, function(res) {
       res.on('data', function(d) {
         //
       });
     }
   );
   req.end();
   

   The full code should look similar to:

   import https from `https`;
   var options = {
       hostname: 'honeycomb.io'
       , path: '/'
       , method: 'GET'
   };
   
   exports.lambdaHandler = async (event, context) => {
     response = {
       'statusCode': 200,
       'body': json.stringify('Hello, World!')
     }
   
     var req = https.request(
       options, function(res) {
         res.on('data', function(d) {
           //
         });
       }
     );  
     req.end();
   
     return response;
   };
   

3. Press [Ctrl] + [s] on your keyboard to save.

4. Select Deploy.

Test Your Implementation 

It’s time to test your implementation and see data in Honeycomb!

1. In the Code tab, select Test.

2. Open Honeycomb and look for a Dataset with the same name as the Lambda function you created in the first step of this guide.

If you see the Dataset, then your data is flowing to Honeycomb!

Creating a Function URL 

To test your Lambda function more easily, you can use a function URL, which is a dedicated HTTP(S) endpoint that you can enable for your Lambda function. You can use the function URL to automate testing.

To create a function URL:

1. In the Lambda console, select Functions from the nav.
2. Locate and select the name of the function for which you you want to create a function URL.
3. Select the Configuration tab, and then select Function URL.
4. Select Create function URL.
5. For Auth type, choose AWS_IAM or NONE, depending on who you would like to have access to your function URL.
6. Select Save.

Propagating Trace Context 

Trace context plays a critical role in linking together individual services into a cohesive trace. In AWS Lambda, ensuring that trace context is correctly passed across multiple Lambda functions is essential for accurate, end-to-end tracing.

Disabling AWS Context Propagation 

By default, AWS Lambda automatically propagates trace context between services. If you need to disable automatic context propagation, you can use either an environment variable or a configuration file.

Disabling Context Propagation Using an Environment Variable 

Set the OTEL_LAMBDA_DISABLE_CONTEXT_PROPAGATION reserved environment variable to true:

OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true

Disabling Context Propagation Using a Configuration File 

Using a configuration file gives you more control over how trace context is managed in your Lambda functions. To disable context propagation using a configuration file:

1. Create a configuration file (in this example, named lambda-config.js), and add the following code:

   // lambda-config.js
   global.configureLambdaInstrumentation = (config) => {
     return {
       ...config,
       disableAwsContextPropagation: true
     }
   }
   

2. Include this configuration file when starting your Lambda function, either as part of the start command or when using the NODE_OPTIONS reserved environment variable:

   NODE_OPTIONS=--require ./lambda-config.js
   

Propagating Context Using Lambda Event Arguments 

When invoking a Lambda function outside of an HTTP context, the trace context may not be included automatically. In these cases, you can use Lambda event arguments to propagate the context manually. For example, in Node.js:

const { trace, context, TraceFlags } = require('@opentelemetry/api');

let tracer = trace.getTracer("aws-lambda-tracer");

exports.handler = async function (event, ctx) {
  let newContext = trace.setSpanContext(context.active(), {
    traceId: event.traceId,
    spanId: event.spanId,
    traceFlags: TraceFlags.SAMPLED,
    isRemote: true });

  context.with(newContext, () => {
    const span = tracer.startSpan('handler function', newContext);
    span.end();
  });

  return context.logStreamName;
};

Troubleshooting 

If you need to troubleshoot, explore these solutions to common issues.

Missing root spans in traces 

If you are using AWS Lambda with API Gateway (or another service that governs traffic), missing root spans for your traces in Honeycomb are likely. This is because API Gateway often generates the initial request but may not propagate tracing headers correctly to your Lambda function.

Some solutions you can try to ensure your traces include root spans include:

• Disable Active Tracing on your Lambda function (in the AWS Console under Configuration > Monitoring).

  Tip

  If you are using Terraform to launch your Lambda function, you can turn off tracing by assigning the Passthrough value to the tracing_config argument. Learn more about the tracing_config argument in Terraform’s documentation for Lambda Function resources.

• To ensure consistent trace context, add the OTEL_PROPAGATORS reserved environment variable to your Lambda function (in the AWS Console under Configuration > Environment variables) and set its value to tracecontext.

• If you call your Lambda function directly from code, make sure the traceparent header is present. With the traceparent header, OpenTelemetry can recognize the request as part of a larger trace and will create a root span. If the traceparent header is missing, add a code block to wrap your entrypoint function in a span. For example, in JavaScript:

  // index.js
  const { trace, context, SpanStatusCode, propagation } = require('@opentelemetry/api');
  
  let tracer = trace.getTracer('tracer.name.here');
  
  exports.handler = async (event) => {
    let span = null;
    if (!event.headers.traceparent) {
      span = tracer.startActiveSpan('something.different', { root: true });
    }
  
    const response = {
      statusCode: 200,
      body: createTrackingId(event.body || 'salt'),
    };
  
    if (span) {
      span.end();
    }
  
    return response;
  };
  

• Ensure your OpenTelemetry Collector is correctly configured to send data to Honeycomb.

• Make sure you have selected an appropriate service.name. By default, the AWS Lambda layer sets service.name to the Lambda function’s name. Because Honeycomb creates a dataset for each service.name, the default AWS Lambda behavior can cause traces from related Lambda functions to be split across different datasets. To keep datasets together, override the default service.name by explicitly setting the OTEL_SERVICE_NAME reserved environment variable for each Lambda function.

Missing non-root spans in traces 

Although the AWS Lambda Setup documentation includes a step that enables Active Tracing, we recommend disabling Active Tracing.

When Active Tracing is enabled, Lambda uses a ParentBased sampler with a sample rate of 5%. The AWS Propagator code always creates a new context, which can cause some spans to be dropped when using a ParentBased sampler. With this sampler, if spans are dropped and DEBUG mode is enabled, you might see the following error in the logs:

DEBUG Recording is off, propagating context in a non-recording span

To resolve this, use an AlwaysOn sampler by setting the OTEL_TRACES_SAMPLER reserved environment variable to always_on:

OTEL_TRACES_SAMPLER=always_on

To confirm the sampler configuration in use, output details to the console:

let tracer = trace.getTracer('<TRACER_NAME_HERE>');
console.log(`Tracer sampler information: ${tracer['_sampler'].toString()}`)

Lambda layer not instrumenting code 

Ensure that you have created the AWS_LAMBDA_EXEC_WRAPPER reserved environment variable, which is essential for initializing OpenTelemetry instrumentation, and have set it to use the OpenTelemetry handler:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit