Send Data with the OpenTelemetry Python SDK

Use the OpenTelemetry Python SDK to instrument Python applications and send telemetry data to Honeycomb. In this guide, you’ll instrument your Python application with the OpenTelemetry Python SDK, including automatic instrumentation.

Before You Begin

Before you can set up automatic instrumentation for your Python application, you will need to do a few things.

Prepare Your Development Environment

To complete the required steps, you will need:

A working Python environment
An application written in Python

Get Your Honeycomb API Key

To send data to Honeycomb, you’ll need to sign up for a free Honeycomb account and create a Honeycomb Ingest API Key. To get started, you can create a key that you expect to swap out when you deploy to production. Name it something helpful, perhaps noting that it’s a getting started key. Make note of your API key; for security reasons, you will not be able to see the key again, and you will need it later!

For setup, make sure you check the “Can create datasets” checkbox so that your data will show up in Honeycomb. Later, when you replace this key with a permanent one, you can uncheck that box.

If you want to use an API key you previously stored in a secure location, you can also look up details for Honeycomb API Keys any time in your Environment Settings, and use them to retrieve keys from your storage location.

Add Automatic Instrumentation

Automatic instrumentation is provided by an agent, a command line tool opentelemetry-instrument that is used to run your application. Add additional instrumentation to your application manually using the included OpenTelemetry Python SDK.

Acquire Dependencies

To add instrumentation, you should install required OpenTelemetry packages and instrumentation libraries.

pip
poetry

Install the OpenTelemetry Python packages:

python -m pip install "opentelemetry-sdk[file-configuration]" \
    opentelemetry-instrumentation \
    opentelemetry-distro \
    opentelemetry-exporter-otlp

The [file-configuration] extra installs the optional dependencies needed to load a YAML configuration file.

Install instrumentation libraries for the packages used by your application. We recommend using the opentelemetry-bootstrap tool that comes with the OpenTelemetry SDK to scan your application packages and print out a list of available instrumentation libraries. You should then add these libraries to your requirements.txt file:
```
opentelemetry-bootstrap >> requirements.txt
pip install -r requirements.txt
```
If you do not use a requirements.txt file, you can install the libraries directly in your current environment:
```
opentelemetry-bootstrap --action=install
```

Install the OpenTelemetry Python packages:

poetry add opentelemetry-instrumentation \
    opentelemetry-distro \
    opentelemetry-exporter-otlp

Install instrumentation libraries for the packages used by your application. We recommend using the opentelemetry-bootstrap tool that comes with the OpenTelemetry SDK to scan your application packages to get a list of available libraries.
```
poetry run opentelemetry-bootstrap
```
The tool does not support installing packages directly when using Poetry, so you must install them manually. For example, to install the Flask instrumentation library:
```
poetry add opentelemetry-instrumentation-flask
```

Configure the OpenTelemetry SDK

Create an otelconfig.yaml file with the following content:

file_format: "1.1"

resource:
  attributes:
    - name: service.name
      value: ${OTEL_SERVICE_NAME:-my-service}
  detection/development:
    detectors:
      - host:
      - os:
      - process:
      - service:

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
            endpoint: https://api.honeycomb.io/v1/traces
            # Use the endpoint below for EU
            # endpoint: https://api.eu1.honeycomb.io/v1/traces
            headers:
              - name: x-honeycomb-team
                value: ${HONEYCOMB_API_KEY}

meter_provider: # (*)
  readers:
    - periodic:
        exporter:
          otlp_http:
            endpoint: https://api.honeycomb.io/v1/metrics
            # Use the endpoint below for EU
            # endpoint: https://api.eu1.honeycomb.io/v1/metrics
            headers:
              - name: x-honeycomb-team
                value: ${HONEYCOMB_API_KEY}
              - name: x-honeycomb-dataset
                value: ${HONEYCOMB_METRICS_DATASET}

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
            endpoint: https://api.honeycomb.io/v1/logs
            # Use the endpoint below for EU
            # endpoint: https://api.eu1.honeycomb.io/v1/logs
            headers:
              - name: x-honeycomb-team
                value: ${HONEYCOMB_API_KEY}

propagator:
  composite:
    - tracecontext:
    - baggage:

Fields marked with an asterisk (*) are required for exporting metrics to Honeycomb. Set the following environment variables before running your application:

Environment Variable	Value
`HONEYCOMB_API_KEY`	Your Honeycomb API key
`OTEL_SERVICE_NAME`	The name of your service
`HONEYCOMB_METRICS_DATASET`	The Honeycomb dataset to send metrics to (*)

If you are using the EU instance, replace https://api.honeycomb.io with https://api.eu1.honeycomb.io in the configuration file.

When OTEL_CONFIG_FILE is set, the configuration file is the single source of truth for the SDK. Other OTEL_* environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using ${VAR_NAME} substitution.

If you use Honeycomb Classic, you must also specify the Dataset for traces using the x-honeycomb-dataset header:

headers:
  - name: x-honeycomb-team
    value: ${HONEYCOMB_API_KEY}
  - name: x-honeycomb-dataset
    value: your-dataset

Enable Python Standard Library Logging

If your application uses Python’s standard library logging module and you want those log records to reach Honeycomb, attach the OpenTelemetry LoggingHandler to the root logger during application startup:

import logging
from opentelemetry._logs import get_logger_provider
from opentelemetry.sdk._logs import LoggingHandler

logging.getLogger().addHandler(
    LoggingHandler(logger_provider=get_logger_provider())
)

This step is a temporary workaround. It will not be required in a future release of the OpenTelemetry Python SDK. Upstream work to honor OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED alongside OTEL_CONFIG_FILE is tracked in open-telemetry/opentelemetry-python#5352.

Run Your Application

  OTEL_CONFIG_FILE=./otelconfig.yaml opentelemetry-instrument python YOUR_APPLICATION_NAME.py

Be sure to replace YOUR_APPLICATION_NAME with the name of your application’s main file. In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.

Add Custom Instrumentation

Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add custom, or manual, instrumentation where appropriate. Follow the instructions below to add custom instrumentation to your code. To learn more about custom, or manual, instrumentation, visit the comprehensive set of topics covered by Manual Instrumentation for Python in OpenTelemetry’s documentation.

Acquire Dependencies

To start adding custom instrumentation, ensure that the opentelemetry-api package exists as a direct dependency in your project. This package provides access to the high-level instrumentation APIs, which gives the ability to retrieve the current span to enrich with additional attributes, to create new spans, and to generate metrics.

  python -m pip install opentelemetry-api

Add Attributes to Spans

Adding context to a currently executing span in a trace can be useful. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. To do this, get the current span from the context and set an attribute with the user ID:

from opentelemetry import trace

# ...

span = trace.get_current_span()
span.set_attribute("user.id", user.id())

Acquire a Tracer

To create spans, you need to acquire a Tracer.

from opentelemetry import trace

tracer = trace.get_tracer("tracer.name.here")

When you create a Tracer, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the Tracer is turned into the library.name field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”. The library.name field is also used with traces created from instrumentation libraries.

Create Spans

To get the full picture of what is happening, you can leverage manual instrumentation to create custom spans that describe what is happening in your application. To do this, create or re-use a Tracer instance and start a span.

from opentelemetry import trace

tracer = trace.get_tracer("tracer.name.here")
with tracer.start_as_current_span("some-long-running-handler"):
    # prepare to auth

    with tracer.start_as_current_span("what-it-takes-to-authorize-handling"):
        # do the auth

    # do the authorized long running handling of cool stuff

Creating Spans Around Methods

You can use a decorator to wrap the execution of a method with a span. The span will be automatically closed once the method has completed.

from opentelemetry import trace

tracer = trace.get_tracer("tracer.name.here")

# ...

@tracer.start_as_current_span("do_work")
def do_work():
    # do some work ...

Add Multi-Span Attributes

Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute, leverage the OpenTelemetry concept of baggage. Baggage allows you to add a key with a value as an attribute to every subsequent child span of the current application context. In Python, baggage is configured as part of the OpenTelemetry trace context. Modifications to the trace context must be both attached and detached when no longer used to make sure context state is disposed of correctly.

Install the opentelemetry-processor-baggage package:

  python -m pip install opentelemetry-processor-baggage

Configure the OpenTelemetry SDK tracer provider to add a BaggageSpanProcessor:

from opentelemetry.processor.baggage import BaggageSpanProcessor, ALLOW_ALL_BAGGAGE_KEYS

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(BaggageSpanProcessor(ALLOW_ALL_BAGGAGE_KEYS))

Add a baggage entry for the current trace. For example,add a user ID attribute to multiple spans in a trace.
```
from opentelemetry import baggage
from opentelemetry.context import attach, detach

...

token = attach(
    baggage.set_baggage("user.id", user.id())
)
with tracer.start_as_current_span(name="hello"):
    span.set_attribute("message", "hello world!")

detach(token)
```
If you do not detach a modified trace context, you will see runtime errors as the OpenTelemetry SDK will detect un-detached contexts in its internal stack.

Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.

Sampling

You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate. Use a TraceIdRatioBased sampler, with a ratio expressed as 1/N. Then, also create a resource attribute called SampleRate with the value of N. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data. In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2"

The value of SampleRate must be a positive integer.

Choosing between gRPC and HTTP

Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind:

Some SDKs default to using gRPC, and it may be easiest to start with the default option.
Some firewall policies are not set up to handle gRPC and require using HTTP.
gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery.

gRPC default export uses port 4317, whereas HTTP default export uses port 4318.

Troubleshooting

To explore common issues when sending data, visit Common Issues with Sending Data in Honeycomb.

​Before You Begin

​Prepare Your Development Environment

​Get Your Honeycomb API Key

​Add Automatic Instrumentation

​Acquire Dependencies

​Configure the OpenTelemetry SDK

​Enable Python Standard Library Logging

​Run Your Application

​Add Custom Instrumentation

​Acquire Dependencies

​Add Attributes to Spans

​Acquire a Tracer

​Create Spans

​Creating Spans Around Methods

​Add Multi-Span Attributes

​Sampling

​Choosing between gRPC and HTTP

​Troubleshooting

Before You Begin

Prepare Your Development Environment

Get Your Honeycomb API Key

Add Automatic Instrumentation

Acquire Dependencies

Configure the OpenTelemetry SDK

Enable Python Standard Library Logging

Run Your Application

Add Custom Instrumentation

Acquire Dependencies

Add Attributes to Spans

Acquire a Tracer

Create Spans

Creating Spans Around Methods

Add Multi-Span Attributes

Sampling

Choosing between gRPC and HTTP

Troubleshooting