If you are affected by the OpenTelemetry HTTP semantic convention stabilization breaking changes, this guide will help you migrate your application from the old semantics to the new, stable semantics.
To identify whether you are affected, visit OpenTelemetry HTTP Semantic Conventions Compatibility.
As a part of the stabilization breaking changes, 17 attributes in the HTTP semantic conventions were renamed, such as http.status_code
changing to http.request.status_code
.
These changes are likely to impact several areas within Honeycomb.
We recommend to review the complete list of attribute changes in OpenTelemetry’s Summary of Changes.
With these attribute changes in mind, we also recommend to review your Honeycomb:
If any of these elements depend on attributes with HTTP semantic conventions, then you will be impacted by these breaking changes.
OpenTelemetry defines a special environment variable, OTEL_SEMCONV_STABILITY_OPT_IN
, that should be added to language instrumentation to help users migrate.
We recommend the following migration steps:
OTEL_SEMCONV_STABILITY_OPT_IN
environment variable to http/dup
so that the instrumentation library will be able to create data using both the old and stable HTTP semantic conventions.OTEL_SEMCONV_STABILITY_OPT_IN
.OTEL_SEMCONV_STABILITY_OPT_IN
to http
so that the instrumentation library only emit the new, stable HTTP semantic conventions.Refer to our language compatibility matrix to determine important library instrumentation versions and whether the library supports http/dup
.
Producing both sets of HTTP semantic conventions at the same time is the best way to ensure a seamless transition between the old and the new HTTP semantic conventions.
Setting the OTEL_SEMCONV_STABILITY_OPT_IN
environment variable to http/dup
enables this dual semantic convention ability for you.
If this is not an option, such as using a language that does not yet support the stable HTTP semantic conventions in OTel, complete these steps (as applicable) to ensure your setup can handle the transition period:
The OpenTelemetry Collector allows for coalescing of data before it reaches Refinery (if applicable) or Honeycomb’s endpoint. We highly recommend that you use an OpenTelemetry Collector to transform and forward your data, especially when dealing with this change.
Use the following OTel Collector configuration to ensure that both new and old span attributes are always present in your data before it gets sent to Refinery or Honeycomb.
Using the configuration below prevents data breakage and allows for incremental updates for other things, like SLOs and Triggers, without risking data loss:
transform/http_semconv_spans:
error_mode: ignore
trace_statements:
- context: span
statements:
# All client and server spans
# rename http.method -> http.request.method
- set(attributes["http.request.method"], attributes["http.method"]) where attributes["http.method"] != nil and attributes["http.request.method"] == nil
# rename http.request.method -> http.method
- set(attributes["http.method"], attributes["http.request.method"]) where attributes["http.method"] == nil and attributes["http.request.method"] != nil
# rename http.status_code -> http.response.status_code
- set(attributes["http.response.status_code"], attributes["http.status_code"]) where attributes["http.status_code"] != nil and attributes["http.response.status_code"] == nil
# rename http.response.status_code -> http.status_code
- set(attributes["http.status_code"], attributes["http.response.status_code"]) where attributes["http.status_code"] == nil and attributes["http.response.status_code"] != nil
# rename browser.user_agent -> user_agent.original
- set(attributes["user_agent.original"], attributes["http.user_agent"]) where attributes["browser.user_agent"] != nil and attributes["user_agent.original"] == nil
# rename user_agent.original -> browser.user_agent
- set(attributes["http.user_agent"], attributes["user_agent.original"]) where attributes["user_agent.original"] != nil and attributes["http.user_agent"] == nil
# rename network.protocol.name -> net.protocol.name
- set(attributes["net.protocol.name"], attributes["network.protocol.name"]) where attributes["network.protocol.name"] != nil and attributes["net.protocol.name"] == nil
# rename net.protocol.name -> network.protocol.name
- set(attributes["network.protocol.name"], attributes["net.protocol.name"]) where attributes["network.protocol.name"] == nil and attributes["net.protocol.name"] != nil
# rename http.flavor -> network.protocol.version
- set(attributes["network.protocol.version"], attributes["http.flavor"]) where attributes["http.flavor"] != nil and attributes["network.protocol.version"] == nil
# rename network.protocol.version -> net.protocol.version
- set(attributes["net.protocol.version"], attributes["network.protocol.version"]) where attributes["net.protocol.version"] == nil and attributes["network.protocol.version"] != nil
# rename network.protocol.version -> http.flavor
- set(attributes["http.flavor"], attributes["network.protocol.version"]) where attributes["http.flavor"] == nil and attributes["network.protocol.version"] != nil
# rename net.protocol.name -> http.flavor
- set(attributes["http.flavor"], attributes["net.protocol.name"]) where attributes["net.protocol.name"] != nil and attributes["http.flavor"] == nil
# rename net.protocol.version -> network.protocol.version
- set(attributes["network.protocol.version"], attributes["net.protocol.version"]) where attributes["net.protocol.version"] != nil and attributes["network.protocol.version"] == nil
# rename net.sock.peer.addr -> network.peer.address
- set(attributes["network.peer.address"], attributes["net.sock.peer.addr"]) where attributes["net.sock.peer.addr"] != nil and attributes["network.peer.address"] == nil
# rename network.peer.address -> net.sock.peer.addr
- set(attributes["net.sock.peer.addr"], attributes["network.peer.address"]) where attributes["net.sock.peer.addr"] == nil and attributes["network.peer.address"] != nil
# rename net.sock.peer.port -> network.peer.port
- set(attributes["network.peer.port"], attributes["net.sock.peer.port"]) where attributes["net.sock.peer.port"] != nil and attributes["network.peer.port"] == nil
# rename network.peer.port -> net.sock.peer.port
- set(attributes["net.sock.peer.port"], attributes["network.peer.port"]) where attributes["net.sock.peer.port"] == nil and attributes["network.peer.port"] != nil
# All client spans
# rename http.url -> url.full
- set(attributes["url.full"], attributes["http.url"]) where attributes["http.url"] != nil and attributes["url.full"] == nil and kind == SPAN_KIND_CLIENT
# rename url.full -> http.url
- set(attributes["http.url"], attributes["url.full"]) where attributes["http.url"] == nil and attributes["url.full"] != nil and kind == SPAN_KIND_CLIENT
# rename http.resend_count -> http.request.resend_count
- set(attributes["http.request.resend_count"], attributes["http.resend_count"]) where attributes["http.resend_count"] != nil and attributes["http.request.resend_count"] == nil and kind == SPAN_KIND_CLIENT
# rename http.request.resend_count -> http.resend_count
- set(attributes["http.resend_count"], attributes["http.request.resend_count"]) where attributes["http.resend_count"] == nil and attributes["http.request.resend_count"] != nil and kind == SPAN_KIND_CLIENT
# rename net.peer.name -> server.address
- set(attributes["server.address"], attributes["net.peer.name"]) where attributes["net.peer.name"] != nil and attributes["server.address"] == nil and kind == SPAN_KIND_CLIENT
# rename server.address -> net.peer.name
- set(attributes["net.peer.name"], attributes["server.address"]) where attributes["net.peer.name"] == nil and attributes["server.address"] != nil and kind == SPAN_KIND_CLIENT
# rename net.peer.port -> server.port
- set(attributes["server.port"], attributes["net.peer.port"]) where attributes["net.peer.port"] != nil and attributes["server.port"] == nil and kind == SPAN_KIND_CLIENT
# rename server.port -> net.peer.port
- set(attributes["net.peer.port"], attributes["server.port"]) where attributes["net.peer.port"] == nil and attributes["server.port"] != nil and kind == SPAN_KIND_CLIENT
# All server spans
# rename http.target -> url.path and url.query
- set(cache["temp"], Split(attributes["http.target"], "?")) where attributes["http.target"] != nil and kind == SPAN_KIND_SERVER
- set(attributes["url.path"], cache["temp"][0]) where attributes["url.path"] == nil and cache["temp"] != nil and Len(cache["temp"]) >= 1 and kind == SPAN_KIND_SERVER
- set(attributes["url.query"], cache["temp"][1]) where attributes["url.query"] == nil and cache["temp"] != nil and Len(cache["temp"]) >= 2 and kind == SPAN_KIND_SERVER
# rename url.path and url.query -> http.target
- set(attributes["http.target"], Concat([attributes["url.path"], attributes["url.query"]], "?")) where attributes["http.target"] == nil and attributes["url.path"] != nil and attributes["url.query"] != nil and kind == SPAN_KIND_SERVER
# rename http.scheme -> url.scheme
- set(attributes["url.scheme"], attributes["http.scheme"]) where attributes["http.scheme"] != nil and attributes["url.scheme"] == nil and kind == SPAN_KIND_SERVER
# rename url.scheme -> http.scheme
- set(attributes["http.scheme"], attributes["url.scheme"]) where attributes["http.scheme"] == nil and attributes["url.scheme"] != nil and kind == SPAN_KIND_SERVER
# rename http.client_ip -> client.address
- set(attributes["client.address"], attributes["http.client_ip"]) where attributes["http.client_ip"] != nil and attributes["client.address"] == nil and kind == SPAN_KIND_SERVER
# rename client.address -> http.client_ip
- set(attributes["http.client_ip"], attributes["client.address"]) where attributes["http.client_ip"] == nil and attributes["client.address"] != nil and kind == SPAN_KIND_SERVER
# rename net.host.name -> server.address
- set(attributes["server.address"], attributes["net.host.name"]) where attributes["net.host.name"] != nil and attributes["server.address"] == nil and kind == SPAN_KIND_SERVER
# rename server.address -> net.host.name
- set(attributes["net.host.name"], attributes["server.address"]) where attributes["net.host.name"] == nil and attributes["server.address"] != nil and kind == SPAN_KIND_SERVER
# rename net.host.port -> server.port
- set(attributes["server.port"], attributes["net.host.port"]) where attributes["net.host.port"] != nil and attributes["server.port"] == nil and kind == SPAN_KIND_SERVER
# rename server.port -> net.host.port
- set(attributes["net.host.port"], attributes["server.port"]) where attributes["net.host.port"] == nil and attributes["server.port"] != nil and kind == SPAN_KIND_SERVER
Coalescing the metrics changes is more complicated, but still possible.
Based on your decision, use one of the two examples below to modify your OTel Collector configuration for metrics.
The following OTel Collector configuration renames old metric values to new metric values:
transform/http_semconv_metrics_new:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
# rename http.method -> http.request.method
- set(attributes["http.request.method"], attributes["http.method"]) where attributes["http.method"] != nil and attributes["http.request.method"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename http.status_code -> http.response.status_code
- set(attributes["http.response.status_code"], attributes["http.status_code"]) where attributes["http.status_code"] != nil and attributes["http.response.status_code"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.peer.name -> server.address
- set(attributes["server.address"], attributes["net.peer.name"]) where attributes["net.peer.name"] != nil and attributes["server.address"] == nil and metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.peer.port -> server.port
- set(attributes["server.port"], attributes["net.peer.port"]) where attributes["net.peer.port"] != nil and attributes["server.port"] == nil and metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.protocol.name -> network.protocol.name
- set(attributes["network.protocol.name"], attributes["net.protocol.name"]) where attributes["net.protocol.name"] != nil and attributes["network.protocol.name"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.protocol.version -> network.protocol.version
- set(attributes["network.protocol.version"], attributes["net.protocol.version"]) where attributes["net.protocol.version"] != nil and attributes["network.protocol.version"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename http.flavor -> network.protocol.version
- set(attributes["network.protocol.version"], attributes["http.flavor"]) where attributes["http.flavor"] != nil and attributes["network.protocol.version"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename http.scheme -> url.scheme
- set(attributes["url.scheme"], attributes["http.scheme"]) where attributes["http.scheme"] != nil and attributes["url.scheme"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.host.name -> server.address
- set(attributes["server.address"], attributes["net.host.name"]) where attributes["net.host.name"] != nil and attributes["server.address"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename net.host.port -> server.port
- set(attributes["server.port"], attributes["net.host.port"]) where attributes["net.host.port"] != nil and attributes["server.port"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename metric, unit and re-aggregate
- set(metric.unit, "s") where (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.unit == "ms" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(value_double, value_double/1000.0) where (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.unit == "s" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(metric.name, "http.client.request.duration") where metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(metric.name, "http.server.request.duration") where metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
The following OTel Collector configuration renames new metric values to old metric values:
transform/http_semconv_metrics_old:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
# rename http.request.method -> http.method
- set(attributes["http.method"], attributes["http.request.method"]) where attributes["http.method"] == nil and attributes["http.request.method"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename http.response.status_code -> http.status_code
- set(attributes["http.status_code"], attributes["http.response.status_code"]) where attributes["http.status_code"] == nil and attributes["http.response.status_code"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename server.address -> net.peer.name
- set(attributes["net.peer.name"], attributes["server.address"]) where attributes["net.peer.name"] == nil and attributes["server.address"] != nil and metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename server.port -> net.peer.port
- set(attributes["net.peer.port"], attributes["server.port"]) where attributes["net.peer.port"] == nil and attributes["server.port"] != nil and metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename network.protocol.name -> net.protocol.name
- set(attributes["net.protocol.name"], attributes["network.protocol.name"]) where attributes["net.protocol.name"] == nil and attributes["network.protocol.name"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename network.protocol.version -> net.protocol.version
- set(attributes["net.protocol.version"], attributes["network.protocol.version"]) where attributes["net.protocol.version"] == nil and attributes["network.protocol.version"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename network.protocol.version -> http.flavor
- set(attributes["http.flavor"], attributes["network.protocol.version"]) where attributes["http.flavor"] == nil and attributes["network.protocol.version"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename url.scheme -> http.scheme
- set(attributes["http.scheme"], attributes["url.scheme"]) where attributes["http.scheme"] == nil and attributes["url.scheme"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename server.address -> net.host.name
- set(attributes["net.host.name"], attributes["server.address"]) where attributes["net.host.name"] == nil and attributes["server.address"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename server.port -> net.host.port
- set(attributes["net.host.port"], attributes["server.port"]) where attributes["net.host.port"] == nil and attributes["server.port"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
# rename metric, unit and re-aggregate
- set(metric.unit, "ms") where (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.unit == "s" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(value_double, value_double*1000.0) where (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.unit == "ms" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(metric.name, "http.client.duration") where metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
- set(metric.name, "http.server.duration") where metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
Refinery, as of version 2.3, can use multiple fields in a condition, which ensures that the same sampling decisions can be made on differently-shaped data. While not a true coalescing since the data is not modified, this feature will help to manage tail sampling when the traces contain spans with both the old and new HTTP semantic conventions.
If you use Refinery, we highly recommend updating Refinery to the latest version and use this capability.
- Name: dynamically sample 200 responses across semantic conventions
Conditions:
- Fields:
- http.status_code
- http.response.status_code
Operator: =
Value: 200
Datatype: int
Fields
is used here instead of Field
.
This configuration ensures that the rule evaluates successfully, regardless of whether data uses http.status_code
or http.response.status_code
as an field.If the OTel Collector is not an option, you can create Derived Columns using the COALESCE
function.
Use COALESCE
to make a new field name that represents two values at once.
It works by taking the value of whichever field it sees first.
To avoid name conflicts in the future, we recommend to name these Derived Columns with a dc.
prefix.
Using the status_code
in an SLI requires embedding the COALESCE
function anywhere the status code is tested or returned, so it can get verbose.