Migrate to Stabilized OpenTelemetry HTTP Semantic Conventions

If you are affected by the OpenTelemetry HTTP semantic convention stabilization breaking changes, this guide will help you migrate your application from the old semantics to the new, stable semantics.

To identify whether you are affected, visit OpenTelemetry HTTP Semantic Conventions Compatibility.

Understand the Impact

As a part of the stabilization breaking changes, 17 attributes in the HTTP semantic conventions were renamed, such as http.status_code changing to http.request.status_code. These changes are likely to impact several areas within Honeycomb.

We recommend to review the complete list of attribute changes in OpenTelemetry’s Summary of Changes.

With these attribute changes in mind, we also recommend to review your Honeycomb:

SLOs
Triggers
Derived Columns
Boards and saved queries
Refinery rule conditions and field lists
OpenTelemetry Collector configurations

If any of these elements depend on attributes with HTTP semantic conventions, then you will be impacted by these breaking changes.

Migration Strategy

OpenTelemetry defines a special environment variable, OTEL_SEMCONV_STABILITY_OPT_IN, that should be added to language instrumentation to help users migrate.

We recommend the following migration steps:

Set the OTEL_SEMCONV_STABILITY_OPT_IN environment variable to http/dup so that the instrumentation library will be able to create data using both the old and stable HTTP semantic conventions.
Update to an instrumentation library version that supports OTEL_SEMCONV_STABILITY_OPT_IN.
Coalesce your data to include the new, stable HTTP semantic conventions.
Change your SLOs, Triggers, Derived Columns, Boards, Refinery Rules, and OpenTelemetry Collector Configurations to use the new, stable HTTP semantic conventions.
Then, set OTEL_SEMCONV_STABILITY_OPT_IN to http so that the instrumentation library only emit the new, stable HTTP semantic conventions.
Test that everything works as expected.
When possible, upgrade to an instrumentation library version that only supports the new, stable HTTP semantic conventions.

Refer to our language compatibility matrix to determine important library instrumentation versions and whether the library supports http/dup.

Coalesce Your Data

Producing both sets of HTTP semantic conventions at the same time is the best way to ensure a seamless transition between the old and the new HTTP semantic conventions. Setting the OTEL_SEMCONV_STABILITY_OPT_IN environment variable to http/dup enables this dual semantic convention ability for you.

If this is not an option, such as using a language that does not yet support the stable HTTP semantic conventions in OTel, complete these steps (as applicable) to ensure your setup can handle the transition period:

Use the OpenTelemetry Collector
Update Refinery Rules
Coalesce using Derived Columns, a non-OTel Collector option

Use the OpenTelemetry Collector

The OpenTelemetry Collector allows for coalescing of data before it reaches Refinery (if applicable) or Honeycomb’s endpoint. We highly recommend that you use an OpenTelemetry Collector to transform and forward your data, especially when dealing with this change.

Coalescing Span Attributes

Use the following OTel Collector configuration to ensure that both new and old span attributes are always present in your data before it gets sent to Refinery or Honeycomb.

Using the configuration below prevents data breakage and allows for incremental updates for other things, like SLOs and Triggers, without risking data loss:

transform/http_semconv_spans:
  error_mode: ignore
  trace_statements:
    - context: span
      statements:
        # All client and server spans
        # rename http.method -> http.request.method
        - set(attributes["http.request.method"], attributes["http.method"]) where attributes["http.method"] != nil and attributes["http.request.method"] == nil

        # rename http.request.method -> http.method
        - set(attributes["http.method"], attributes["http.request.method"]) where attributes["http.method"] == nil and attributes["http.request.method"] != nil

        # rename http.status_code -> http.response.status_code
        - set(attributes["http.response.status_code"], attributes["http.status_code"]) where attributes["http.status_code"] != nil and attributes["http.response.status_code"] == nil

        # rename http.response.status_code -> http.status_code
        - set(attributes["http.status_code"], attributes["http.response.status_code"]) where attributes["http.status_code"] == nil and attributes["http.response.status_code"] != nil

        # rename browser.user_agent -> user_agent.original
        - set(attributes["user_agent.original"], attributes["http.user_agent"]) where attributes["browser.user_agent"] != nil and attributes["user_agent.original"] == nil

        # rename user_agent.original -> browser.user_agent
        - set(attributes["http.user_agent"], attributes["user_agent.original"]) where attributes["user_agent.original"] != nil and attributes["http.user_agent"] == nil

        # rename network.protocol.name -> net.protocol.name
        - set(attributes["net.protocol.name"], attributes["network.protocol.name"]) where attributes["network.protocol.name"] != nil and attributes["net.protocol.name"] == nil

        # rename net.protocol.name -> network.protocol.name
        - set(attributes["network.protocol.name"], attributes["net.protocol.name"]) where attributes["network.protocol.name"] == nil and attributes["net.protocol.name"] != nil

        # rename http.flavor -> network.protocol.version
        - set(attributes["network.protocol.version"], attributes["http.flavor"]) where attributes["http.flavor"] != nil and attributes["network.protocol.version"] == nil

        # rename network.protocol.version -> net.protocol.version
        - set(attributes["net.protocol.version"], attributes["network.protocol.version"]) where attributes["net.protocol.version"] == nil and attributes["network.protocol.version"] != nil

        # rename network.protocol.version -> http.flavor 
        - set(attributes["http.flavor"], attributes["network.protocol.version"]) where attributes["http.flavor"] == nil and attributes["network.protocol.version"] != nil

        # rename net.protocol.name -> http.flavor
        - set(attributes["http.flavor"], attributes["net.protocol.name"]) where attributes["net.protocol.name"] != nil and attributes["http.flavor"] == nil

        # rename net.protocol.version -> network.protocol.version
        - set(attributes["network.protocol.version"], attributes["net.protocol.version"]) where attributes["net.protocol.version"] != nil and attributes["network.protocol.version"] == nil

        # rename net.sock.peer.addr -> network.peer.address
        - set(attributes["network.peer.address"], attributes["net.sock.peer.addr"]) where attributes["net.sock.peer.addr"] != nil and attributes["network.peer.address"] == nil

        # rename network.peer.address -> net.sock.peer.addr
        - set(attributes["net.sock.peer.addr"], attributes["network.peer.address"]) where attributes["net.sock.peer.addr"] == nil and attributes["network.peer.address"] != nil

        # rename net.sock.peer.port -> network.peer.port
        - set(attributes["network.peer.port"], attributes["net.sock.peer.port"]) where attributes["net.sock.peer.port"] != nil and attributes["network.peer.port"] == nil

        # rename network.peer.port -> net.sock.peer.port
        - set(attributes["net.sock.peer.port"], attributes["network.peer.port"]) where attributes["net.sock.peer.port"] == nil and attributes["network.peer.port"] != nil

        # All client spans
        # rename http.url -> url.full
        - set(attributes["url.full"], attributes["http.url"]) where attributes["http.url"] != nil and attributes["url.full"] == nil and kind == SPAN_KIND_CLIENT

        # rename url.full -> http.url
        - set(attributes["http.url"], attributes["url.full"]) where attributes["http.url"] == nil and attributes["url.full"] != nil and kind == SPAN_KIND_CLIENT

        # rename http.resend_count -> http.request.resend_count
        - set(attributes["http.request.resend_count"], attributes["http.resend_count"]) where attributes["http.resend_count"] != nil and attributes["http.request.resend_count"] == nil and kind == SPAN_KIND_CLIENT

        # rename http.request.resend_count -> http.resend_count
        - set(attributes["http.resend_count"], attributes["http.request.resend_count"]) where attributes["http.resend_count"] == nil and attributes["http.request.resend_count"] != nil and kind == SPAN_KIND_CLIENT
          
        # rename net.peer.name -> server.address
        - set(attributes["server.address"], attributes["net.peer.name"]) where attributes["net.peer.name"] != nil and attributes["server.address"] == nil and kind == SPAN_KIND_CLIENT

        # rename server.address -> net.peer.name
        - set(attributes["net.peer.name"], attributes["server.address"]) where attributes["net.peer.name"] == nil and attributes["server.address"] != nil and kind == SPAN_KIND_CLIENT

        # rename net.peer.port -> server.port
        - set(attributes["server.port"], attributes["net.peer.port"]) where attributes["net.peer.port"] != nil and attributes["server.port"] == nil and kind == SPAN_KIND_CLIENT

        # rename server.port -> net.peer.port
        - set(attributes["net.peer.port"], attributes["server.port"]) where attributes["net.peer.port"] == nil and attributes["server.port"] != nil and kind == SPAN_KIND_CLIENT

        # All server spans
        # rename http.target -> url.path and url.query
        - set(cache["temp"], Split(attributes["http.target"], "?")) where attributes["http.target"] != nil and kind == SPAN_KIND_SERVER
        - set(attributes["url.path"], cache["temp"][0]) where attributes["url.path"] == nil and cache["temp"] != nil and Len(cache["temp"]) >= 1 and kind == SPAN_KIND_SERVER
        - set(attributes["url.query"], cache["temp"][1]) where attributes["url.query"] == nil and cache["temp"] != nil and Len(cache["temp"]) >= 2 and kind == SPAN_KIND_SERVER

        # rename url.path and url.query -> http.target
        - set(attributes["http.target"], Concat([attributes["url.path"], attributes["url.query"]], "?")) where attributes["http.target"] == nil and attributes["url.path"] != nil and attributes["url.query"] != nil and kind == SPAN_KIND_SERVER

        # rename http.scheme -> url.scheme
        - set(attributes["url.scheme"], attributes["http.scheme"]) where attributes["http.scheme"] != nil and attributes["url.scheme"] == nil and kind == SPAN_KIND_SERVER

        # rename url.scheme -> http.scheme
        - set(attributes["http.scheme"], attributes["url.scheme"]) where attributes["http.scheme"] == nil and attributes["url.scheme"] != nil and kind == SPAN_KIND_SERVER

        # rename http.client_ip -> client.address
        - set(attributes["client.address"], attributes["http.client_ip"]) where attributes["http.client_ip"] != nil and attributes["client.address"] == nil and kind == SPAN_KIND_SERVER

        # rename client.address ->  http.client_ip
        - set(attributes["http.client_ip"], attributes["client.address"]) where attributes["http.client_ip"] == nil and attributes["client.address"] != nil and kind == SPAN_KIND_SERVER

        # rename net.host.name -> server.address
        - set(attributes["server.address"], attributes["net.host.name"]) where attributes["net.host.name"] != nil and attributes["server.address"] == nil and kind == SPAN_KIND_SERVER

        # rename server.address -> net.host.name 
        - set(attributes["net.host.name"], attributes["server.address"]) where attributes["net.host.name"] == nil and attributes["server.address"] != nil and kind == SPAN_KIND_SERVER

        # rename net.host.port -> server.port
        - set(attributes["server.port"], attributes["net.host.port"]) where attributes["net.host.port"] != nil and attributes["server.port"] == nil and kind == SPAN_KIND_SERVER

        # rename server.port -> net.host.port
        - set(attributes["net.host.port"], attributes["server.port"]) where attributes["net.host.port"] == nil and attributes["server.port"] != nil and kind == SPAN_KIND_SERVER

Coalescing Metrics

Coalescing the metrics changes is more complicated, but still possible.

Tip

If you use metrics, determine whether you want the metrics to appear in Honeycomb with the old values or the new values.

Based on your decision, use one of the two examples below to modify your OTel Collector configuration for metrics.

The following OTel Collector configuration renames old metric values to new metric values:

transform/http_semconv_metrics_new:
  error_mode: ignore
  metric_statements:
    - context: datapoint
      statements:
        # rename http.method -> http.request.method
        - set(attributes["http.request.method"], attributes["http.method"]) where attributes["http.method"] != nil and attributes["http.request.method"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        
        # rename http.status_code -> http.response.status_code
        - set(attributes["http.response.status_code"], attributes["http.status_code"]) where attributes["http.status_code"] != nil and attributes["http.response.status_code"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        
        # rename net.peer.name -> server.address
        - set(attributes["server.address"], attributes["net.peer.name"]) where attributes["net.peer.name"] != nil and attributes["server.address"] == nil and metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename net.peer.port -> server.port
        - set(attributes["server.port"], attributes["net.peer.port"]) where attributes["net.peer.port"] != nil and attributes["server.port"] == nil and metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename net.protocol.name -> network.protocol.name
        - set(attributes["network.protocol.name"], attributes["net.protocol.name"]) where attributes["net.protocol.name"] != nil and attributes["network.protocol.name"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename net.protocol.version -> network.protocol.version
        - set(attributes["network.protocol.version"], attributes["net.protocol.version"]) where attributes["net.protocol.version"] != nil and attributes["network.protocol.version"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename http.flavor -> network.protocol.version
        - set(attributes["network.protocol.version"], attributes["http.flavor"]) where attributes["http.flavor"] != nil and attributes["network.protocol.version"] == nil and (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename http.scheme -> url.scheme
        - set(attributes["url.scheme"], attributes["http.scheme"]) where attributes["http.scheme"] != nil and attributes["url.scheme"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename net.host.name -> server.address
        - set(attributes["server.address"], attributes["net.host.name"]) where attributes["net.host.name"] != nil and attributes["server.address"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename net.host.port -> server.port
        - set(attributes["server.port"], attributes["net.host.port"]) where attributes["net.host.port"] != nil and attributes["server.port"] == nil and metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename metric, unit and re-aggregate
        - set(metric.unit, "s") where (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.unit == "ms" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(value_double, value_double/1000.0) where (metric.name == "http.client.duration" or metric.name == "http.server.duration") and metric.unit == "s" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(metric.name, "http.client.request.duration") where metric.name == "http.client.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(metric.name, "http.server.request.duration") where metric.name == "http.server.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

The following OTel Collector configuration renames new metric values to old metric values:

transform/http_semconv_metrics_old:
  error_mode: ignore
  metric_statements:
    - context: datapoint
      statements:
        # rename http.request.method -> http.method
        - set(attributes["http.method"], attributes["http.request.method"]) where attributes["http.method"] == nil and attributes["http.request.method"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        
        # rename http.response.status_code -> http.status_code
        - set(attributes["http.status_code"], attributes["http.response.status_code"]) where attributes["http.status_code"] == nil and attributes["http.response.status_code"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        
        # rename server.address -> net.peer.name 
        - set(attributes["net.peer.name"], attributes["server.address"]) where attributes["net.peer.name"] == nil and attributes["server.address"] != nil and metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename server.port -> net.peer.port 
        - set(attributes["net.peer.port"], attributes["server.port"]) where attributes["net.peer.port"] == nil and attributes["server.port"] != nil and metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename network.protocol.name -> net.protocol.name
        - set(attributes["net.protocol.name"], attributes["network.protocol.name"]) where attributes["net.protocol.name"] == nil and attributes["network.protocol.name"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename network.protocol.version -> net.protocol.version 
        - set(attributes["net.protocol.version"], attributes["network.protocol.version"]) where attributes["net.protocol.version"] == nil and attributes["network.protocol.version"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename network.protocol.version -> http.flavor  
        - set(attributes["http.flavor"], attributes["network.protocol.version"]) where attributes["http.flavor"] == nil and attributes["network.protocol.version"] != nil and (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename url.scheme -> http.scheme 
        - set(attributes["http.scheme"], attributes["url.scheme"]) where attributes["http.scheme"] == nil and attributes["url.scheme"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename server.address -> net.host.name
        - set(attributes["net.host.name"], attributes["server.address"]) where attributes["net.host.name"] == nil and attributes["server.address"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename server.port -> net.host.port
        - set(attributes["net.host.port"], attributes["server.port"]) where attributes["net.host.port"] == nil and attributes["server.port"] != nil and metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

        # rename metric, unit and re-aggregate
        - set(metric.unit, "ms") where (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.unit == "s" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(value_double, value_double*1000.0) where (metric.name == "http.client.request.duration" or metric.name == "http.server.request.duration") and metric.unit == "ms" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(metric.name, "http.client.duration") where metric.name == "http.client.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM
        - set(metric.name, "http.server.duration") where metric.name == "http.server.request.duration" and metric.type == METRIC_DATA_TYPE_HISTOGRAM

Update Refinery Rules

Refinery, as of version 2.3, can use multiple fields in a condition, which ensures that the same sampling decisions can be made on differently-shaped data. While not a true coalescing since the data is not modified, this feature will help to manage tail sampling when the traces contain spans with both the old and new HTTP semantic conventions.

If you use Refinery, we highly recommend updating Refinery to the latest version and use this capability.

- Name: dynamically sample 200 responses across semantic conventions
  Conditions:
    - Fields:
        - http.status_code
        - http.response.status_code
      Operator: =
      Value: 200
      Datatype: int

Note

Fields is used here instead of Field. This configuration ensures that the rule evaluates successfully, regardless of whether data uses http.status_code or http.response.status_code as an field.

Coalesce using Derived Columns

If the OTel Collector is not an option, you can create Derived Columns using the COALESCE function.

Screenshot of a derived column using coalesce

Use COALESCE to make a new field name that represents two values at once.
It works by taking the value of whichever field it sees first. To avoid name conflicts in the future, we recommend to name these Derived Columns with a dc. prefix.

Using the status_code in an SLI requires embedding the COALESCE function anywhere the status code is tested or returned, so it can get verbose.