Getting Service Mesh and API Gateway Data into Honeycomb

The world of service meshes and API gateways opens new possibilities for collecting tracing data from your production infrastructure.

While the data from service meshes and API gateways cannot match the level of detail and granularity that can be achieved through code level instrumentation, it can be a good first step along the observability journey.

Most, if not all, service meshes and API gateways have distributed tracing capabilities baked in, allowing telemetry data to be sent to Honeycomb with relative ease.

Service Meshes

Istio

Istio supports tracing out of the box using either the Zipkin or Jaeger format.

To send the data to Honeycomb, install the OpenTelemetry Collector. Once installed, configure the Collector to receive the Jaeger or Zipkin data and export it to Honeycomb using the OTLP format.

Next, follow Istio’s distributed tracing documentation. Make sure to configure the global.tracer.zipkin.address setting to point at the OpenTelemetry Collector.

To properly trace all services in your system, ensure that you are forwarding tracing headers from your apps so that Istio can inject the correct tracing information as requests are made and received. See this section on trace context propagation in the Istio docs for details on how to do this.

Here is an example configuration to update MeshConfig that may be passed in as a separate file with the install command istioctl install --set profile=demo -y -f ./tracing.yaml:

# tracing.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        zipkin:
          address: otel-collector.default:9411
        sampling: 100
---

An alternative option is to use the OpenCensus Agent, which includes OpenTelemetry-compatible W3C trace context, and this removes the need to add B3 propagation if it is not already being used.

spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        openCensusAgent:
            address: otel-collector.default:55678
        sampling: 100
---

Then, the OpenTelemetry Collector needs to include an OpenCensus receiver in the receivers and service/pipelines sections:

# otel-collector.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: otel-collector
data:
  otel-collector-config: |
    receivers:

      otlp:
        protocols:
          grpc:
          http:
      opencensus:
        endpoint: 0.0.0.0:55678

    processors:
      batch:

    exporters:

      otlphttp:
        endpoint: "https://api.honeycomb.io"
        headers:
          "x-honeycomb-team": "<API_KEY>"

    service:
      pipelines:
        traces:
          receivers: [otlp,opencensus]
          processors: [batch]
          exporters: [otlphttp]    
---

AWS App Mesh

AWS App Mesh supports tracing out of the box using the Jaeger format.

To send the data to Honeycomb, install the OpenTelemetry Collector. Once installed, configure the Collector to receive the Jaeger data and export it to Honeycomb using the OTLP format.

Next, follow App Mesh’s Jaeger documentation. Make sure to configure the tracing.address and tracing.port settings to point at the OpenTelemetry Collector.

API Gateways

Ambassador

Ambassador supports tracing out of the box using the Zipkin format.

To send the data to Honeycomb, install the OpenTelemetry Collector. Once installed, configure the Collector to receive the Zipkin data and export it to Honeycomb using the OTLP format.

Next, follow Ambassador’s Zipkin documentation, and make sure to configure the service setting to point at the OpenTelemetry Collector.

Kong

Kong supports tracing out of the box using the Zipkin format.