Kubernetes and Honeycomb

K8s Agent Configuration

Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications' logs, cluster logs, and resource metrics so that you can answer all the questions you have.

The Honeycomb agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like:

How did response time change after a canary deployment?
How does application performance vary with container resource limits?
Are application errors happening on specific nodes, or across the fleet?

The Honeycomb agent can also collect resource and status metrics from nodes, pods, containers, and volumes. This data answers questions like:

Which pods are using the most CPU?
How does container resource use compare to limits?
What do system metrics look like at the node level?
Why are pods failing to start?

Requirements 🔗

To use the Helm package, you will need Helm version 3 or better. You can still install the agent using kubectl though configuration may be more involved. You wil also need your Honeycomb API key.

Install using either Helm, or kubectl directly.

Getting started using Helm 🔗

By default, this Helm chart will collect metrics from all nodes and pods, and watchers configuration to capture logs from the following system components:

kube-controller-manager
kube-scheduler

To install the agent using the Helm package manager:

Add the Honeycomb Helm repository

helm repo add honeycomb https://honeycombio.github.io/helm-charts

Install the agent

helm install honeycomb honeycomb/honeycomb --set honeycomb.apiKey=YOUR_API_KEY

Note: See the Helm chart for more details on advanced configuration options.

Getting started using kubectl 🔗

To install the agent using kubectl:

Create a honeycomb namespace

kubectl create namespace honeycomb

Store your Honeycomb API Key as a Kubernetes secret

kubectl create secret generic honeycomb --namespace honeycomb \
       --from-literal=api-key=YOUR_API_KEY

Add the agent to your cluster

kubectl apply -f https://raw.githubusercontent.com/honeycombio/honeycomb-kubernetes-agent/main/examples/quickstart.yaml

Logs collection 🔗

The agent will collect logs based on configured watchers. These watchers will match pods based on the configured selection and parse the logs for the pod using the specified parser. The Helm chart exposes a watchers property that you can use. If installed using kubectl, you will need to modify the watchers section of the ConfigMap.

Using modified log watchers configuration 🔗

A configuration to collect and parse logs from the Kubernetes controller manager and scheduler would look like this:

watchers:
  - dataset: kubernetes-logs
    labelSelector: component=kube-controller-manager
    namespace: kube-system
    parser: glog
  - <dataset: kubernetes-logs
    labelSelector: component=kube-scheduler
    namespace: kube-system
    parser: glog

Capturing events with Heptio Eventrouter 🔗

An optional configuration is to capture Kubernetes events using the Heptio Eventrouter component. Configure the Eventrouter to use the json sink for logs, and you can capture them with this watchers configuration

watchers:
  - dataset: k8s-eventrouter
    labelSelector: app=eventrouter
    namespace: olly
    parser: json
    processors:
      - drop_field:
          field: old_event

Note: You’ll want to configure the agent to parse logs from your applications. But that depends a lot on your applications! Read more about customizing the agent.

Metrics collection 🔗

The agent collects metrics from nodes, pods, containers, and volumes. For each collection interval, an event for each resource will be sent to Honeycomb that contains all collected metrics as fields to the event. Kubernetes labels for the resources will be added as fields to the event with a label. prefix. Pods and containers also get additional status and restart metrics. The default collection interval is every 10 seconds, for nodes and pods resources only. Follow details on how to configure the agent to modifying your configuration accordingly.

Metrics list 🔗

Metrics collected depend on the Kubernetes resource type. You can filter metrics on the k8s.resource.type field.

metric	node	pod	container	volume
metrics.cpu.usage	x	x	x
metrics.cpu.utilization	x	x	x
metrics.filesystem.available	x	x	x
metrics.filesystem.capacity	x	x	x
metrics.filesystem.usage	x	x	x
metrics.memory.available	x	x	x
metrics.memory.major_page_faults	x	x	x
metrics.memory.page_faults	x	x	x
metrics.memory.rss	x	x	x
metrics.memory.usage	x	x	x
metrics.memory.utilization	x	x	x
metrics.memory.working_set	x	x	x
metrics.network.bytes.receive	x	x
metrics.network.bytes.send	x	x
metrics.network.errors.receive	x	x
metrics.network.errors.send	x	x
metrics.uptime	x	x	x
metrics.volume.available				x
metrics.volume.capacity				x
metrics.volume.inodes.free				x
metrics.volume.inodes.total				x
metrics.volume.inodes.used				x
metrics.volume.used				x
status.exitcode			x
status.message		x	x
status.phase		x
status.ready			x
status.reason		x	x
status.restart		x	x
status.restart_count		x	x
status.restart_delta		x	x
status.state			x

Modifying configuration 🔗

You can modify the agent’s configuration via the ConfigMap deployed. By default, Kubernetes resource metrics and logs from the controller and scheduler will be collected. Follow details on how to configure the agent to modifying your configuration accordingly.

Getting help 🔗

For any questions or difficulties, don’t hesitate to get in touch at support@honeycomb.io, or use the chat widget in the lower right.