We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Kubernetes and Honeycomb


Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications' logs, cluster logs, and resource metrics so that you can answer all the questions you have.

Overview of Kubernetes integration

The Honeycomb agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like:

The Honeycomb agent can also collect resource and status metrics from nodes, pods, containers, and volumes. This data answers questions like:

Requirements  🔗

To use the Helm package, you will need Helm version 3 or better. You can still install the agent using kubectl though configuration may be more involved. You wil also need your Honeycomb API key.

Install using either Helm, or kubectl directly.

Getting started using Helm  🔗

By default, this Helm chart will collect metrics from all nodes and pods, and watchers configuration to capture logs from the following system components:

To install the agent using the Helm package manager:

  1. Add the Honeycomb Helm repository
helm repo add honeycomb https://honeycombio.github.io/helm-charts
  1. Install the agent
helm install honeycomb honeycomb/honeycomb --set honeycomb.apiKey=YOUR_API_KEY

Note: See the Helm chart for more details on advanced configuration options.

Getting started using kubectl  🔗

To install the agent using kubectl:

  1. Store your Honeycomb API Key as a Kubernetes secret
kubectl create secret generic -n default honeycomb \
       --from-literal=api-key=YOUR_API_KEY
  1. Add the agent to your cluster
kubectl apply -f https://raw.githubusercontent.com/honeycombio/honeycomb-kubernetes-agent/main/examples/quickstart.yaml

Logs collection  🔗

The agent will collect logs based on configured watchers. These watchers will match pods based on the configured selection and parse the logs for the pod using the specified parser. The Helm chart exposes a watchers property that you can use. If installed using kubectl, you will need to modify the watchers section of the ConfigMap.

Using modified log watchers configuration  🔗

A configuration to collect and parse logs from the Kubernetes controller manager and scheduler would look like this:

watchers:
  - dataset: kubernetes-logs
    labelSelector: component=kube-controller-manager
    namespace: kube-system
    parser: glog
  - <dataset: kubernetes-logs
    labelSelector: component=kube-scheduler
    namespace: kube-system
    parser: glog

Capturing events with Heptio Eventrouter  🔗

An optional configuration is to capture Kubernetes events using the Heptio Eventrouter component. Configure the Eventrouter to use the json sink for logs, and you can capture them with this watchers configuration

watchers:
  - dataset: k8s-eventrouter
    labelSelector: app=eventrouter
    namespace: olly
    parser: json
    processors:
      - drop_field:
          field: old_event

Note: You’ll want to configure the agent to parse logs from your applications. But that depends a lot on your applications! Read more about customizing the agent.

Metrics collection  🔗

The agent collects metrics from nodes, pods, containers, and volumes. For each collection interval, an event for each resource will be sent to Honeycomb that contains all collected metrics as fields to the event. Kubernetes labels for the resources will be added as fields to the event with a label. prefix. Pods and containers also get additional status and restart metrics. The default collection interval is every 10 seconds, for nodes and pods resources only. Follow details on how to configure the agent to modifying your configuration accordingly.

Metrics list  🔗

Metrics collected depend on the Kubernetes resource type. You can filter metrics on the k8s.resource.type field.

metric node pod container volume
metrics.cpu.usage x x x
metrics.cpu.utilization x x x
metrics.filesystem.available x x x
metrics.filesystem.capacity x x x
metrics.filesystem.usage x x x
metrics.memory.available x x x
metrics.memory.major_page_faults x x x
metrics.memory.page_faults x x x
metrics.memory.rss x x x
metrics.memory.usage x x x
metrics.memory.utilization x x x
metrics.memory.working_set x x x
metrics.network.bytes.receive x x
metrics.network.bytes.send x x
metrics.network.errors.receive x x
metrics.network.errors.send x x
metrics.uptime x x x
metrics.volume.available x
metrics.volume.capacity x
metrics.volume.inodes.free x
metrics.volume.inodes.total x
metrics.volume.inodes.used x
metrics.volume.used x
status.exitcode x
status.message x x
status.phase x
status.ready x
status.reason x x
status.restart x x
status.restart_count x x
status.restart_delta x x
status.state x

Modifying configuration  🔗

You can modify the agent’s configuration via the ConfigMap deployed. By default, Kubernetes resource metrics and logs from the controller and scheduler will be collected. Follow details on how to configure the agent to modifying your configuration accordingly.

Getting help  🔗

For any questions or difficulties, don’t hesitate to get in touch at support@honeycomb.io, or use the chat widget in the lower right.