With Honeycomb and OpenTelemetry, you can analyze and query data across not only your applications, but also your entire Kubernetes cluster. The following components available in the OpenTelemetry Collector will allow you to gather the best data from your Kubernetes cluster and applications to be analyzed in Honeycomb.
OpenTelemetry’s Kubernetes Attributes Processor automatically discovers Kubernetes pods, extracts their metadata (for example, pod name or node name), and adds the extracted metadata to spans, metrics, and logs as resource attributes.
Because the Kubernetes Attributes Processor adds Kubernetes context to your telemetry, you can correlate your application’s traces, metrics, and logs with your Kubernetes telemetry, such as pod metrics and events. We highly recommend that you include this processor in any Collector that receives telemetry from Kubernetes pods.
OpenTelemetry’s Kubeletstats Receiver gathers metrics about the Kubernetes node and its workloads. This receiver can gather metrics like container memory usage, pod cpu usage, and node network errors. All of the gathered telemetry includes Kubernetes metadata, such as pod name or node name.
When you use the Kubeletstats Receiver with the Kubernetes Attributes Processor to enhance your application telemetry, you can correlate your application traces, metrics, and logs with the metrics produced by the Kubeletstats Receiver.
OpenTelemetry Collectors use the Kubernetes Cluster Receiver to collect metrics about the state of the cluster as a whole. This receiver can gather metrics about node conditions, pod phases, container restarts, available and desired deployments, and more. Since the receiver gathers telemetry for the cluster as a whole, only one instance of the receiver is needed across the cluster in order to collect all the data.
OpenTelemetry’s Kubernetes Objects Receiver collects objects from the Kubernetes API server. Because the receiver gathers telemetry for the cluster as a whole, only one instance of the receiver is needed across the cluster in order to collect all the data. Most commonly, this receiver is used to watch Kubernetes Events.
To learn more about using the Kubernetes Objects Receiver to send Kubernetes Events to Honeycomb, visit Kubernetes Events with the OpenTelemetry Collector.
OpenTelemetry’s Filelog Receiver tails and parses logs from files. Although the Filelog Receiver is not Kubernetes specific, we recommend it as the primary solution for collecting any logs from Kubernetes.
The Filelog Receiver processes logs through a series of operators that are chained together, each of which performs a simple responsibility, such as parsing JSON.
Configuring a Filelog Receiver can be complicated.
The easiest way to get started is to use the OpenTelemetry Collector Helm chart with the logsCollection preset.
The OpenTelemetry Collector’s Marker Exporter allows you to send a Honeycomb Marker based on the shape of incoming telemetry.
In your OpenTelemetry Collector exporter configuration, you can specify a set of rules that will be evaluated against incoming telemetry.
When a condition is met, a marker is sent and appears in Honeycomb.
For example, whenever a Kubernetes Event contains a reason of Backoff, the configuration below sends a marker:
exporters:
honeycombmarker:
api_key: YOUR-API-KEY-HERE
markers:
# Creates a new marker each time the exporter sees a Kubernetes event with a reason of Backoff
- type: k8s-backoff-events
rules:
- log_conditions:
- IsMap(body) and IsMap(body["object"]) and body["object"]["reason"] == "Backoff"