Kubernetes and Honeycomb

Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications’ logs, cluster events, cluster resource and state metrics, so that you can answer all the questions you have.

The Honeycomb agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like:
- How did response time change after a canary deployment?
- How does application performance vary with container resource limits?
- Are application errors happening on specific nodes, or across the fleet?
Heapster is an open-source project that collects pod and node-level resource metrics, plus cluster events. This data answers questions like:
- Which pods are using the most CPU?
- How does container resource use compare to limits?
- What do system metrics look like at the node level?
- Why are pods failing to start?

To get the integrations up and running in your own cluster, follow the instructions below. For examples of using Honeycomb to answer questions about your cluster and applications, see the use cases guide.

Getting started

If you haven’t already, sign up for a free Honeycomb trial account. Then store your API key as a Kubernetes secret:

kubectl create secret generic -n default honeycomb-writekey \
    --from-literal=key=YOUR_API_KEY

You are currently logged in to the team, so we have populated the write key here to the first write key for that team.

Then follow the steps below to deploy the integrations in a cluster.

Note: If you’re deploying the integrations on Google Container Engine (GKE), you’ll have to grant your current Google identity cluster-admin role before creating any ClusterRoles used by the Honeycomb integrations:

kubectl create clusterrolebinding myname-cluster-admin-binding \
    --clusterrole=cluster-admin --user=<my_current_google_identity@example.org>

See the GKE documentation for more details.

The manifests for the Honeycomb integrations are also available on Github. If you’d like to run the integrations differently, they can easily be modified to fit your needs.

Sending cluster events and resource metrics

Run Heapster with a Honeycomb sink:

kubectl create -f https://honeycomb.io/download/kubernetes/metrics/honeycomb-heapster.yaml

Check the status of your new Heapster pod

kubectl --namespace=kube-system get pods -l k8s-app=heapster

# NAME                                          READY     STATUS    RESTARTS   AGE
# heapster-1760086053-11dnv                     1/1       Running   0          7h

Head over to the Honeycomb UI and query the kubernetes-cluster-events dataset for event logs and the kubernetes-resource-metrics dataset for resource metrics.

Sending application logs

First, configure and run the agent as a DaemonSet. Kubernetes implementations are slightly different, so choose the quickstart configuration that is appropriate to your environment:

Google Kubernetes Engine

kubectl create -f https://honeycomb.io/download/kubernetes/logs/quickstart-gke.yaml

Microsoft AKS

kubectl create -f https://honeycomb.io/download/kubernetes/logs/quickstart-aks.yaml

Amazon EKS

kubectl create -f https://honeycomb.io/download/kubernetes/logs/quickstart-aws-eks.yaml

Generic Kubernetes Cluster

kubectl create -f https://honeycomb.io/download/kubernetes/logs/quickstart.yaml

Next, check the status of your new pods:

kubectl get daemonset

# NAME                   DESIRED   CURRENT   READY     UP-TO-DATE   ...
# honeycomb-agent-v1.1   1         1         1         1            ...

Finally, head over to the Honeycomb UI and query the kubernetes-logs dataset.

Note: You’ll want to configure the agent to parse logs from your applications. But that depends a lot on your applications! Read more about customizing the agent.

Getting help

For any questions or difficulties, don’t hesitate to get in touch at support@honeycomb.io, or use the chat widget in the lower right.

The source code for the agent and deployment manifests is also open-source and available on Github. Issues or patches are welcome!