Export Data with AWS Lambda Layer + OpenTelemetry

If you are using OpenTelemetry, an official AWS-managed OpenTelemetry Lambda Layer lets you use OpenTelemetry in your Lambda Functions and export data asynchronously from those functions.

The AWS-managed OpenTelemetry Lambda Layer works by embedding a stripped-down version of the OpenTelemetry (OTel) Collector inside an AWS Lambda Extension Layer. To use it in any language, configure an OTLP exporter to send to the OTel Collector in the Lambda Layer.

AWS maintains pre-configured Lambda Layers for several languages with automatic instrumentation:

Java
Python
Nodejs
.NET
Go

See AWS’s documentation on Getting Started with AWS Lambda Layers for more information.

If not using any of the above listed languages, or would prefer to manually build and configure a layer, refer to AWS’s documentation on Manual Steps for Private Lambda Layers.

Fix missing root spans in traces

If using Lambda with API Gateway or another service that governs traffic, missing root spans for your traces in Honeycomb are likely. To mitigate, complete these steps:

Disable AWS X-Ray tracing on your Lambda in the AWS UI (Configuration > Monitoring).

If using Terraform to launch your lambda function, you can turn off tracing with the tracing_config option PassThrough. Learn more here.
Add an environment variable named OTEL_PROPAGATORS with value tracecontext to your Lambda in the AWS UI (Configuration > Environment variables).
Optional: If you call your Lambda directly from code and the traceparent header does not exist, add a code block to wrap your initial entrypoint function with a span. Otherwise, skip this step. For example, if using JavaScript:

// index.js
const { trace, context, SpanStatusCode, propagation } = require('@opentelemetry/api');

let tracer = trace.getTracer('tracer.name.here');

exports.handler = async (event) => {
  let span = null;
  if (!event.headers.traceparent) {
    span = tracer.startActiveSpan('something.different', { root: true });
  }

  const response = {
    statusCode: 200,
    body: createTrackingId(event.body || 'salt'),
  };

  if (span) {
    span.end();
  }

  return response;
};

Ensure your OpenTelemetry Collector configuration is correctly configured to send to Honeycomb.
Ensure a good service.name for your Lambdas. By default, the AWS Lambda Layer uses the Lambda name for the service.name resource attribute. If you want to keep datasets together for related lambdas, this default behavior can cause problems. Override the default service name by defining the OTEL_SERVICE_NAME environment variable for each Lambda.

After following the above steps, traces in Honeycomb should no longer have missing root spans.

Fix missing other spans in traces

AWS Lambda Setup docs include a step to enable Active Tracing for automatic instrumentation. Our step above to fix missing root spans suggest disabling Active Tracing. When Active tracing is enabled in a Lambda function, it defaults to a ParentBased Sampler, with a sample rate of 5%. The AWS Propagator code always creates a new context, thus resulting in some spans not being recorded when using a ParentBased Sampler.

When spans are dropped with this sampler and DEBUG is enabled, the following error may be logged:

DEBUG Recording is off, propagating context in a non-recording span

To solve this, set the sampler to an AlwaysOn sampler using the OTEL_TRACES_SAMPLER environment variable:

OTEL_TRACES_SAMPLER=always_on

One way to confirm the sampler configuration in use is to output the details to the console:

let tracer = trace.getTracer('tracer.name.here');
console.log(`Tracer sampler information: ${tracer['_sampler'].toString()}`)

Lambda Layer is Not Instrumenting Code at All

The environment variable AWS_LAMBDA_EXEC_WRAPPER is required as it initializes OpenTelemetry.

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler

Propagating Trace Context

Disable AWS Context Propagation

AWS context propagation is enabled by default. Use an environment variable or a configuration file to disable.

To disable with an environment variable:

OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION=true

To disable with a configuration file:

Add the following code to your configuration file.

// lambda-config.js
global.configureLambdaInstrumentation = (config) => {
  return {
    ...config,
    disableAwsContextPropagation: true
  }
}

Include the configuration file with your application as part of the start command or using the NODE_OPTIONS environment variable:
```
NODE_OPTIONS=--require ./lambda-config.js
```

Propagate Context Using Lambda Event Arguments

If a lambda is not invoked via HTTP, you can use event arguments to propagate context:

const { trace, context, TraceFlags } = require('@opentelemetry/api');

let tracer = trace.getTracer("aws-lambda-tracer");

exports.handler = async function (event, ctx) {
  let newContext = trace.setSpanContext(context.active(), {
    traceId: event.traceId,
    spanId: event.spanId,
    traceFlags: TraceFlags.SAMPLED,
    isRemote: true });

  context.with(newContext, () => {
    const span = tracer.startSpan('handler function', newContext);
    span.end();
  });

  return context.logStreamName;
};

Honeycomb.io Documentation