OpenTelemetry for Python | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

OpenTelemetry for Python

This guide will help you add OpenTelemetry to your web service, show you how to add additional custom context to that instrumentation, and ensure that instrumentation data is being sent to Honeycomb.

Requirements  🔗

These instructions will explain how to set up manual instrumentation for a service written in Python. In order to follow along, you will need:

  • Python 3.6 or higher
  • An app written in Python
  • A Honeycomb API Key, which you can find on your Team Settings page. If you don’t have an API key yet, sign up for a free Honeycomb account.

Adding Instrumentation  🔗

To instrument your Python app with OpenTelemetry, you will need to install these packages:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-exporter-otlp-proto-grpc

Next, we need to create and configure a tracer instance. Here we’ll use some OpenTelemetry components to describe our app and set up an exporter that will send spans to Honeycomb.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from grpc import ssl_channel_credentials

# resource describes app-level information that will be added to all spans
resource = Resource(attributes={
    "service.name": os.environ.get("SERVICE_NAME")
})

# create new trace provider with our resource
trace_provider = TracerProvider(resource=resource)

# create exporter to send spans to Honeycomb
otlp_exporter = OTLPSpanExporter(
    endpoint="api.honeycomb.io:443",
    insecure=False,
    credentials=ssl_channel_credentials(),
    headers=(
        ("x-honeycomb-team", os.environ.get("HONEYCOMB_API_KEY")),
        ("x-honeycomb-dataset", os.environ.get("HONEYCOMB_DATASET"))
    )
)

# register exporter with provider
trace_provider.add_span_processor(
    BatchSpanProcessor(otlp_exporter)
)

# register trace provider
trace.set_tracer_provider(trace_provider)

Note that we’re setting a few configuration options using environment variables. You can set these as inline environment variables like this when starting your python app:

HONEYCOMB_API_KEY=my-api-key \
HONEYCOMB_DATASET=my-dataset \
SERVICE_NAME=my-favorite-service \
python app.py

Creating Spans  🔗

Now we have a tracer configured, we can create spans to describe what is happening in your application. This could be a HTTP handler, a long running operation, a database fetch, etc. Spans are created in a parent-child pattern, so each time you create a new span, the current active span is used as it’s parent.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("http-handler"):
    with tracer.start_as_current_span("my-cool-function"):
        # do something

Adding Context to Spans  🔗

It is often beneficial to add context to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID:

from opentelemetry import trace

...

span = trace.get_current_span()
span.set_attribute("user_id", user.id())

Augmenting or Scrubbing Spans  🔗

Sometimes you want to make sure that certain information does not escape your application or service. This could be for regulatory reasons regarding personally identifiable information, or you want to ensure that certain information does not end being stored with a vendor. Scrubbing attributes from a span is possible with a custom span processor. Span processors in OpenTelemetry allow you to hook into the lifecycle of a span and modify the span contents before sending to a backend.

The example below creates and registers a span processor that removes potentially sensitive information as spans finish but before they are exported.

class ScrubbingSpanProcessor(trace.SpanProcessor):
    def on_start(
        self, span: Span, parent_context: typing.Optional[Context] = None
    ) -> None:
        # blank out any attriubtes that are sensitive
        span.set_attribute("forbidden", "")

# register span processor with trace provider during app initialization
trace.get_tracer_provider().add_span_processor(ScrubbingSpanProcessor)

Sampling  🔗

To control how many spans are being sent to Honeycomb, you can configure a sampler to send a portion of your traffic. The sampler is configured with a sample rate, where 1/X traces are selected to be exported and all spans in a selected trace are sent to Honeycomb.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

...

# sampling sends a portion of generated spans based on a 1/X population size
sample_rate = int(os.environ.get("SAMPLE_RATE"))
sampler = TraceIdRatioBased(1/sample_rate)

# create new trace provider with our resource and sampler
trace_provider = TracerProvider(
    sampler=sampler
)

# register trace provider
trace.set_tracer_provider(trace_provider)

Distributed Trace Propagation  🔗

When a service calls another service, you want to be sure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Trace context propagation is done by sending and parsing headers that conform to the W3C Trace Context specification.

In order for this to work, both the sending and receiving service must be using the same propagation format, and both services must be configured to send data to the same Honeycomb dataset.