Kubernetes and Honeycomb | Honeycomb

We use cookies or similar technologies to personalize your online experience & tailor marketing to you. Many of our product features require cookies to function properly.

Read our privacy policy I accept cookies from this site

Honeycomb.io Docs
  1. Docs
  2. Send Data to Honeycomb
  3. Other Integrations
  4. Kubernetes

Kubernetes and Honeycomb

Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications' logs, cluster logs, and resource metrics so that you can answer all the questions you have.

Overview of Kubernetes integration

The Honeycomb agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like:

  • How did response time change after a canary deployment?
  • How does application performance vary with container resource limits?
  • Are application errors happening on specific nodes, or across the fleet?

The Honeycomb agent can also collect resource and status metrics from nodes, pods, containers, and volumes. This data answers questions like:

  • Which pods are using the most CPU?
  • How does container resource use compare to limits?
  • What do system metrics look like at the node level?
  • Why are pods failing to start?

Requirements 

To use the Helm package, you will need Helm version 3 or better. You can still install the agent using kubectl though configuration may be more involved. You wil also need your Honeycomb API key.

Install using either Helm, or kubectl directly.

Getting Started Using Helm 

By default, this Helm chart will collect metrics from all nodes and pods, and watchers configuration to capture logs from the following system components:

  • kube-controller-manager
  • kube-scheduler

To install the agent using the Helm package manager:

  1. Add the Honeycomb Helm repository
helm repo add honeycomb https://honeycombio.github.io/helm-charts
  1. Install the agent
helm install honeycomb honeycomb/honeycomb --set honeycomb.apiKey=YOUR_API_KEY

Note: See the Helm chart for more details on advanced configuration options.

Getting Started Using Kubectl 

To install the agent using kubectl:

  1. Create a honeycomb namespace
kubectl create namespace honeycomb
  1. Store your Honeycomb API Key as a Kubernetes secret
kubectl create secret generic honeycomb --namespace honeycomb \
       --from-literal=api-key=YOUR_API_KEY
  1. Add the agent to your cluster
kubectl apply -f https://raw.githubusercontent.com/honeycombio/honeycomb-kubernetes-agent/main/examples/quickstart.yaml

Logs Collection 

The agent will collect logs based on configured watchers. These watchers will match pods based on the configured selection and parse the logs for the pod using the specified parser. The Helm chart exposes a watchers property that you can use. If installed using kubectl, you will need to modify the watchers section of the ConfigMap.

Using modified log watchers configuration 

A configuration to collect and parse logs from the Kubernetes controller manager and scheduler would look like this:

watchers:
  - dataset: kubernetes-logs
    labelSelector: component=kube-controller-manager
    namespace: kube-system
    parser: glog
  - <dataset: kubernetes-logs
    labelSelector: component=kube-scheduler
    namespace: kube-system
    parser: glog

Capturing events with Heptio Eventrouter 

An optional configuration is to capture Kubernetes events using the Heptio Eventrouter component. Configure the Eventrouter to use the json sink for logs, and you can capture them with this watchers configuration:

watchers:
  - dataset: k8s-eventrouter
    labelSelector: app=eventrouter
    namespace: olly
    parser: json
    processors:
      - drop_field:
          field: old_event

Note: You will want to configure the agent to parse logs from your applications, but that depends a lot on your applications! Read more about customizing the agent.

Metrics Collection 

The agent collects metrics from nodes, pods, containers, and volumes. For each collection interval, an event for each resource will be sent to Honeycomb that contains all collected metrics as fields to the event. Kubernetes labels for the resources will be added as fields to the event with a label. prefix. Pods and containers also get additional status and restart metrics. The default collection interval is every 10 seconds, for nodes and pods resources only. Follow details on how to configure the agent to modifying your configuration accordingly.

Metrics List 

Metrics collected depend on the Kubernetes resource type. You can filter metrics on the k8s.resource.type field.

metric node pod container volume
metrics.cpu.usage x x x
metrics.cpu.utilization x x x
metrics.filesystem.available x x x
metrics.filesystem.capacity x x x
metrics.filesystem.usage x x x
metrics.memory.available x x x
metrics.memory.major_page_faults x x x
metrics.memory.page_faults x x x
metrics.memory.rss x x x
metrics.memory.usage x x x
metrics.memory.utilization x x x
metrics.memory.working_set x x x
metrics.network.bytes.receive x x
metrics.network.bytes.send x x
metrics.network.errors.receive x x
metrics.network.errors.send x x
metrics.uptime x x x
metrics.volume.available x
metrics.volume.capacity x
metrics.volume.inodes.free x
metrics.volume.inodes.total x
metrics.volume.inodes.used x
metrics.volume.used x
status.exitcode x
status.message x x
status.phase x
status.ready x
status.reason x x
status.restart x x
status.restart_count x x
status.restart_delta x x
status.state x

Modifying Configuration 

You can modify the agent’s configuration via the ConfigMap deployed. By default, Kubernetes resource metrics and logs from the controller and scheduler will be collected. Follow details on how to configure the agent to modifying your configuration accordingly.

Getting Help 

For any questions or difficulties, do not hesitate to get in touch at support@honeycomb.io, or use the chat widget in the lower right.

Did you find what you were looking for?

Table of Contents