Send Events with the Kubernetes Objects Receiver

If you are running applications that are not using OpenTelemetry in your Kubernetes cluster, you can still collect the events from your cluster.

In this guide, you will learn how to use an OpenTelemetry Collector to get additional insight into your data by sending your Kubernetes Events to Honeycomb using OpenTelemetry’s Kubernetes Objects Receiver.

Before You Begin 

Before beginning this guide, you should have:

Collect Events 

Enable the Kubernetes Objects Receiver to collect events by adding the kubernetesEvents preset in the values file for your OpenTelemetry Deployment-mode Collector. Place it near the top of the values file under the config section:

presets:
  kubernetesEvents:
    enabled: true
Warning
Apply this to only a Deployment-mode Collector with a replica count of 1. Running multiple receivers on the same Kubernetes cluster will cause duplicate events to be sent for every Collector running this receiver.

Format Event Data 

By default, the kubernetesEvents preset in the OpenTelemetry Helm chart will configure the Collector to pull all of the events from the Kubernetes cluster and export them as logs.

Honeycomb users can derive a great deal of value from these logs–when they are structured appropriately. To transform the bodies of these logs into structured content that is easily queried in Honeycomb, use the OpenTelemetry Transform Processor to parse the event data.

To force the event data to conform to the standard k8s.* attribute naming that all other telemetry types use:

  1. Add the following to the values file for your OpenTelemetry Deployment-mode Collector. Place it near the top of the values file under the config section:

      processors:
        transform/events:
          error_mode: ignore
          log_statements:
            - context: log
              statements:
                - set(attributes["watch-type"], body["type"]) where IsMap(body) and body["type"] != nil
    
                - merge_maps(attributes, body, "upsert") where IsMap(body) and body["object"] == nil
                - merge_maps(attributes, body["object"], "upsert") where IsMap(body) and body["object"] != nil
    
                - merge_maps(attributes, attributes[ "metadata"], "upsert") where IsMap(attributes[ "metadata"])
    
                # Maps the name of the resource to the right k8s.* attribute
                - set(attributes["k8s.pod.name"], attributes["regarding"] ["name"]) where attributes["regarding"]["kind"] == "Pod"
                - set(attributes["k8s.node.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Node"
                - set(attributes["k8s.job.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Job"
                - set(attributes["k8s.cronjob.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "CronJob"
                - set(attributes["k8s.namespace.name"], attributes["regarding"]["namespace"]) where attributes["regarding"]["kind"] == "Pod" or attributes["regarding"]["kind"] == "Job" or attributes["regarding"]["kind"] == "CronJob"
    
                # Converts event types to Otel log Severities
                - set(severity_text, attributes["type"]) where attributes["type"] == "Normal" or attributes["type"] == "Warning"
                - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["type"] == "Normal"
                - set(severity_number, SEVERITY_NUMBER_WARN) where attributes["type"] == "Warning"
    
  2. Under the config section, update your pipelines section to match the following:

      service:
        pipelines:
          metrics:
            exporters: [ otlp/k8s-metrics ]
          logs:
            processors: [ memory_limiter, transform/events, batch ]
            exporters: [ otlp/k8s-logs ]