Understanding the behavior of applications running on Kubernetes can be daunting. Honeycomb integrates with Kubernetes to collect your applications' logs, cluster logs, and resource metrics so that you can answer all the questions you have.
The Honeycomb agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like:
The Honeycomb agent can also collect resource and status metrics from nodes, pods, containers, and volumes. This data answers questions like:
To use the Helm package, you will need Helm version 3 or better.
You can still install the agent using kubectl
though configuration may be more involved.
You wil also need your Honeycomb API key.
Install using either Helm, or kubectl directly.
By default, this Helm chart will collect metrics from all nodes and pods, and watchers configuration to capture logs from the following system components:
To install the agent using the Helm package manager:
helm repo add honeycomb https://honeycombio.github.io/helm-charts
helm install honeycomb honeycomb/honeycomb --set honeycomb.apiKey=YOUR_API_KEY
Note: See the Helm chart for more details on advanced configuration options.
To install the agent using kubectl:
honeycomb
namespacekubectl create namespace honeycomb
kubectl create secret generic honeycomb --namespace honeycomb \
--from-literal=api-key=YOUR_API_KEY
kubectl apply -f https://raw.githubusercontent.com/honeycombio/honeycomb-kubernetes-agent/main/examples/quickstart.yaml
The agent will collect logs based on configured watchers
.
These watchers
will match pods based on the configured selection and parse the logs for the pod using the specified parser.
The Helm chart exposes a watchers
property that you can use.
If installed using kubectl, you will need to modify the watchers
section of the ConfigMap.
A configuration to collect and parse logs from the Kubernetes controller manager and scheduler would look like this:
watchers:
- dataset: kubernetes-logs
labelSelector: component=kube-controller-manager
namespace: kube-system
parser: glog
- <dataset: kubernetes-logs
labelSelector: component=kube-scheduler
namespace: kube-system
parser: glog
An optional configuration is to capture Kubernetes events using the Heptio Eventrouter component.
Configure the Eventrouter to use the json
sink for logs, and you can capture them with this watchers
configuration:
watchers:
- dataset: k8s-eventrouter
labelSelector: app=eventrouter
namespace: olly
parser: json
processors:
- drop_field:
field: old_event
Note: You will want to configure the agent to parse logs from your applications, but that depends a lot on your applications! Read more about customizing the agent.
The agent collects metrics from nodes, pods, containers, and volumes.
For each collection interval, an event for each resource will be sent to Honeycomb that contains all collected metrics as fields to the event.
Kubernetes labels for the resources will be added as fields to the event with a label.
prefix.
Pods and containers also get additional status and restart metrics.
The default collection interval is every 10 seconds, for nodes and pods resources only.
Follow details on how to configure the agent to modifying your configuration accordingly.
Metrics collected depend on the Kubernetes resource type.
You can filter metrics on the k8s.resource.type
field.
metric | node | pod | container | volume |
---|---|---|---|---|
metrics.cpu.usage | x | x | x | |
metrics.cpu.utilization | x | x | x | |
metrics.filesystem.available | x | x | x | |
metrics.filesystem.capacity | x | x | x | |
metrics.filesystem.usage | x | x | x | |
metrics.memory.available | x | x | x | |
metrics.memory.major_page_faults | x | x | x | |
metrics.memory.page_faults | x | x | x | |
metrics.memory.rss | x | x | x | |
metrics.memory.usage | x | x | x | |
metrics.memory.utilization | x | x | x | |
metrics.memory.working_set | x | x | x | |
metrics.network.bytes.receive | x | x | ||
metrics.network.bytes.send | x | x | ||
metrics.network.errors.receive | x | x | ||
metrics.network.errors.send | x | x | ||
metrics.uptime | x | x | x | |
metrics.volume.available | x | |||
metrics.volume.capacity | x | |||
metrics.volume.inodes.free | x | |||
metrics.volume.inodes.total | x | |||
metrics.volume.inodes.used | x | |||
metrics.volume.used | x | |||
status.exitcode | x | |||
status.message | x | x | ||
status.phase | x | |||
status.ready | x | |||
status.reason | x | x | ||
status.restart | x | x | ||
status.restart_count | x | x | ||
status.restart_delta | x | x | ||
status.state | x |
You can modify the agent’s configuration via the ConfigMap
deployed.
By default, Kubernetes resource metrics and logs from the controller and scheduler will be collected.
Follow details on how to configure the agent to modifying your configuration accordingly.
For any questions or difficulties, join our Pollinators Community Slack to ask questions and learn more.
Did you find what you were looking for?