Honeycomb has a OpenTelemetry Distribution for Python to instrument your applications and send telemetry data to Honeycomb.
Automatic instrumentation is provided by an agent, a command line tool opentelemetry-instrument
that is used to run your application.
You can add additional instrumentation to your application manually using the included OpenTelemetry Python SDK.
Honeycomb’s OpenTelemetry Distribution for Python gives you all of the capabilities provided by the generic OpenTelemetry Distribution for Python and also allows you to:
These instructions will explain how to set up automatic and manual instrumentation for a service written in Python. To follow along, you will need:
There are several examples that configure applications to send OpenTelemetry data to Honeycomb.
Install packages:
python -m pip install honeycomb-opentelemetry --pre
opentelemetry-bootstrap
tool that comes with the OpenTelemetry Python SDK to scan your application packages and generate the list of available libraries.
By default, the tool prints out the instrumentation libraries that are available, which you can add to your requirements.txt
file:opentelemetry-bootstrap >> requirements.txt
pip install -r requirements.txt
If you do not use a requirements.txt
file, you can install the libraries directly in your current environment:
opentelemetry-bootstrap --action=install
Install packages:
poetry add honeycomb-opentelemetry
opentelemetry-bootstrap
tool that comes with the OpenTelemetry Python SDK to scan your application packages and generate the list of available libraries.poetry run opentelemetry-bootstrap
The tool does not support installing packages directly when using Poetry, so instead you will need to install them manually. For example, to install the Flask instrumentation library:
poetry add opentelemetry-instrumentation-flask
Use environment variables to configure the Honeycomb OpenTelemetry Python distribution package:
export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"
Variable | Description |
---|---|
OTEL_SERVICE_NAME |
Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. |
HONEYCOMB_API_KEY |
Your API Key generated in Honeycomb. Learn how to find your Honeycomb API Key. |
Use environment variables to configure the Honeycomb OpenTelemetry Python distribution package:
export OTEL_SERVICE_NAME="your-service-name"
export HONEYCOMB_API_KEY="your-api-key"
Variable | Description |
---|---|
OTEL_SERVICE_NAME |
Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. |
HONEYCOMB_API_KEY |
Your API Key generated in Honeycomb. Learn how to find your Honeycomb API Key. |
If you are a Honeycomb Classic user, you must also specify the Dataset using the HONEYCOMB_TRACES_DATASET
environment variable.
A Dataset is a bucket where data gets stored in Honeycomb.
export HONEYCOMB_TRACES_DATASET=my-traces
This is the complete list of configuration options for the Honeycomb Distribution.
Environment Variable | Default Value | Description |
---|---|---|
HONEYCOMB_API_KEY |
None | [required – see note below] Your Honeycomb API key |
OTEL_SERVICE_NAME |
unknown_service |
[required – see note below] service.name attribute, where all trace data is sent |
HONEYCOMB_TRACES_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending traces |
HONEYCOMB_METRICS_APIKEY |
Value of HONEYCOMB_API_KEY |
Your Honeycomb API key for sending metrics |
HONEYCOMB_METRICS_DATASET |
None | Honeycomb dataset where metrics will be sent |
HONEYCOMB_API_ENDPOINT |
api.honeycomb.io:443 |
Honeycomb ingest endpoint |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for traces (defaults to the value of HONEYCOMB_API_ENDPOINT ) |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT |
Value of HONEYCOMB_API_ENDPOINT |
Honeycomb ingest endpoint for metrics (defaults to the value of HONEYCOMB_API_ENDPOINT ) |
SAMPLE_RATE |
1 (retain all data) |
Sample rate for the deterministic sampler. Must be a positive integer. |
HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS |
false |
Enable local visualizations |
DEBUG |
false |
Enable debug mode |
Run your Python application using the OpenTelemetry Python automatic instrumentation tool opentelemetry-instrument
, which configures the OpenTelemetry SDK:
opentelemetry-instrument python YOUR_APPLICATION_NAME.py
Be sure to replace YOUR_APPLICATION_NAME
with the name of your application’s main file.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
Run your Python application using the OpenTelemetry Python automatic instrumentation tool opentelemetry-instrument
, which configures the OpenTelemetry SDK:
poetry run opentelemetry-instrument python YOUR_APPLICATION_NAME.py
Be sure to replace YOUR_APPLICATION_NAME
with the name of your application’s main file.
In Honeycomb’s UI, you should now see your application’s incoming requests and outgoing HTTP calls generate traces.
By default, the Honeycomb OpenTelemetry SDK uses the gRPC protocol to send telemetry data. To use HTTP instead of gRPC, set the export protocol for the OpenTelemetry SDK using the following environment variable:
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add manual instrumentation where appropriate.
To start adding manual instrumentation, ensure that the opentelemetry-api
package exists as a direct dependency in your project.
This package provides access to the high-level instrumentation APIs, which gives the ability to retrieve the current span to enrich with additional attributes, to create new spans, and to generate metrics.
python -m pip install opentelemetry-api
poetry add opentelemetry-api
To create spans, your application needs a Tracer
.
from opentelemetry import trace
tracer = trace.get_tracer("tracer.name.here")
When you create a Tracer
, OpenTelemetry requires you to give it a name as a string.
This string is the only required parameter.
When traces are sent to Honeycomb, the name of the Tracer
is turned into the library.name
field, which can be used to show all spans created from a particular tracer.
In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different “layers” of your application, then use the name that corresponds to that “layer”.
The library.name
field is also used with traces created from instrumentation libraries.
It is often beneficial to add context to a currently executing span in a trace. For example, you may have an application or service that handles users and you want to associate the user with the span when querying your dataset in Honeycomb. In order to do this, get the current span from the context and set an attribute with the user ID:
from opentelemetry import trace
# ...
span = trace.get_current_span()
span.set_attribute("user.id", user.id())
Automatic instrumentation can show the shape of requests to your system, but only you know the really important parts.
In order to get the full picture of what’s happening, you will have to add manual instrumentation and create some custom spans.
To do this, create or re-use a Tracer
instance and start a span.
from opentelemetry import trace
tracer = trace.get_tracer("tracer.name.here")
with tracer.start_as_current_span("some-long-running-handler"):
# prepare to auth
with tracer.start_as_current_span("what-it-takes-to-authorize-handling"):
# do the auth
# do the authorized long running handling of cool stuff
You can also use a decorator to wrap the execution of a method with a span. The span will be automatically closed once the method has completed.
from opentelemetry import trace
tracer = trace.get_tracer("tracer.name.here")
# ...
@tracer.start_as_current_span("do_work")
def do_work():
# do some work ...
Sometimes you will want to add the same attribute to many spans within the same trace. These attributes may include variables calculated during your program, or other useful values for correlation or debugging purposes.
We will leverage the OpenTelemetry concept of baggage to do this.
Baggage allows you to add a key
with a value
as an attribute to every subsequent child span of the current application context.
In Python, baggage is configured as part of the OpenTelemetry trace context.
Modifications to the trace context must be both attached and detached when no longer used to make sure context state is disposed of correctly.
The example below shows how you can add a user ID attribute to multiple spans in a trace.
from opentelemetry import baggage
from opentelemetry.context import attach, detach
...
token = attach(
baggage.set_baggage("user.id", user.id())
)
with tracer.start_as_current_span(name="hello"):
span.set_attribute("message", "hello world!")
detach(token)
Note: Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do NOT put sensitive information in the Baggage attributes.
The OpenTelemetry documentation for Python has a comprehensive set of topics on manual instrumentation.
To control how many spans are being sent to Honeycomb, you can configure the OpenTelemetry SDK to sample the data it generates. Configuration of sampling can be done with environment variables. The following example sets a sample rate of 2, meaning 50% of traces are sampled and sent to Honeycomb:
export SAMPLE_RATE="2"
If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like Honeycomb Refinery, or when you have more robust sampling needs.
When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.
Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.
Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.
The primary purpose of Honeycomb’s Distribution for Python is to streamline configuration and to instrument as quickly and easily as possible. Under the hood, the Honeycomb Distribution is using OpenTelemetry for Python, which means OpenTelemetry can be used with or without this Distribution. It may be unnecessary for advanced users or those already instrumented with OpenTelemetry to use the Distribution.
The Honeycomb Distribution reads specific variables and translates them to variables understood by the upstream OpenTelemetry SDK.
For example, the distribution automatically configures exporters to send telemetry data to Honeycomb’s API api.honeycomb.io
.
Therefore, to send data to Honeycomb using OpenTelemetry without the Distribution, a different configuration is necessary to match expected variables.
Follow the directions below to instrument with OpenTelemetry for Python using the Python SDK.
First, install the opentelemetry-distro and exporter packages.
python -m pip install opentelemetry-distro \
opentelemetry-exporter-otlp
Second, install instrumentation libraries for the packages used by your application.
The recommended way is to use the opentelemetry-bootstrap
tool that comes with the OpenTelemetry SDK to scan your application packages to get the list of available libraries.
By default, the tool prints out the instrumentation libraries that are available, which you can add to your requirements.txt
file:
opentelemetry-bootstrap >> requirements.txt
pip install -r requirements.txt
If you do not use a requirements.txt
file, you can install the libraries directly in your current environment:
opentelemetry-bootstrap --action=install
First, install the opentelemetry-distro and exporter packages.
poetry add opentelemetry-distro \
opentelemetry-exporter-otlp
Second, install instrumentation libraries for the packages used by your application.
The recommended way is to use the opentelemetry-bootstrap
tool that comes with the OpenTelemetry SDK to scan your application packages to get the list of available libraries.
poetry run opentelemetry-bootstrap
The tool does not support installing packages directly when using Poetry, so instead you will need to install them manually. For example, to install the Flask instrumentation library:
poetry add opentelemetry-instrumentation-flask
Configure OpenTelemetry to send events to Honeycomb using environment variables.
The header x-honeycomb-team
is your API key.
Your service name will be used as the Service Dataset in Honeycomb, which is where data is stored.
The service name is specified by OTEL_SERVICE_NAME
.
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
export OTEL_SERVICE_NAME="your-service-name"
Now you can run your application using the OpenTelemetry Python automatic instrumentation tool opentelemetry-instrument
, which configures the OpenTelemetry SDK, and see traces for your app:
opentelemetry-instrument python myapp.py
Configure OpenTelemetry to send events to Honeycomb using environment variables.
The header x-honeycomb-team
is your API key.
Your service name will be used as the Service Dataset in Honeycomb, which is where data is stored.
The service name is specified by OTEL_SERVICE_NAME
.
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
export OTEL_SERVICE_NAME="your-service-name"
Now you can run your application using the OpenTelemetry Python automatic instrumentation tool opentelemetry-instrument
, which configures the OpenTelemetry SDK, and see traces for your app:
poetry run opentelemetry-instrument python myapp.py
If you are a Honeycomb Classic user, the Dataset also must be specified using the HONEYCOMB_TRACES_DATASET
environment variable.
A Dataset is a bucket where data gets stored in Honeycomb.
export HONEYCOMB_TRACES_DATASET=my-traces
More details on configuration options can be found in OpenTelemetry docs under Agent Configuration.
You can configure the OpenTelemetry SDK to sample the data it generates. Honeycomb weights sampled data based on sample rate, so you must set a resource attribute containing the sample rate.
Use a TraceIdRatioBased
sampler, with a ratio expressed as 1/N
.
Then, also create a resource attribute called SampleRate
with the value of N
.
This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data.
In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5).
export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG=0.5
export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2"
The value of SampleRate
must be a positive integer.
If a This trace has multiple spans sharing the same non-null span ID
error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once.
One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the Tracing API throughout the application runtime to add manual instrumentation.
Using MyPy requires turning on support for namespace packages.
To turn on support from the command line, run:
mypy --namespace-packages
Or to turn on support from your project configuration file, add:
[tool.mypy]
namespace_packages = true
The OpenTelemetry Python SDK typically shows errors in the console when applicable. If no errors appear but your data is not in Honeycomb as expected, you can enable debug mode, which prints all spans to the console. This will help confirm whether your app is being instrumented with the data you expect.
Set the DEBUG
environment variable:
export DEBUG=true
Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes.
If the application uses Flask, instrumentation will not work if Flask debugging is enabled.
Unset the FLASK_DEBUG
variable or set it to false
.
If DEBUG
is enabled and FLASK_DEBUG
is disabled, the output in the console will show:
DEBUG:sitecustomize:Instrumented flask
* Debug mode: off
If the application uses Django, instrumentation requires the DJANGO_SETTINGS_MODULE
and the --noreload
flag.
First set the environment variable for DJANGO_SETTINGS_MODULE
based on the name of your settings file.
export DJANGO_SETTINGS_MODULE=myapp.settings
Then run the application with the --noreload
flag to avoid Django running main
twice.
For example, the command to run the application may look like this:
opentelemetry-instrument python manage.py runserver --noreload
The service name is a required configuration value.
If it is unspecified, all trace data will be sent to a default dataset called unknown_service
.
Honeycomb’s OpenTelemetry Distribution for Python can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation.
To enable local visualizations, set the HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS
environment variable to true
:
export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true
Then, run your application:
opentelemetry-instrument python myapp.py
The output displays the name of the root span and a link to Honeycomb that shows its trace. For example:
Trace for <root-span-name>
Honeycomb link: <link to Honeycomb trace>
Select the link to view the trace in detail within the Honeycomb UI.
Honeycomb supports receiving telemetry data via OpenTelemetry’s native protocol, OTLP, over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP protobuf definitions are 0.7.0 for traces and metrics.
If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK’s protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK.
You may receive a 464
error response from the Honeycomb API when sending telemetry using gRPC and HTTP1.
The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers.