1. Docs
  2. Send Data to Honeycomb
  3. Kubernetes
  4. Create Telemetry Pipeline

Create a Telemetry Pipeline

Whether you want to maximize Honeycomb’s capabilities or begin with a more limited set of telemetry data for your Kubernetes applications, we recommend that you use OpenTelemetry, a highly configurable, open-source, and vendor-neutral instrumentation framework.

In this guide, you will learn how to get answers about how your applications on Kubernetes are performing in production using OpenTelemetry Collectors and Honeycomb–and you’ll do it in under 10 minutes.

When you finish, you’ll have visibility into in-depth Kubernetes data, including Kubernetes logs, events, and node/cluster metrics. And you’ll be given the opportunity to take the next step toward leveraging Honeycomb’s full potential by instrumenting your code.

Overview 

In the next 10 minutes, you will create a series of OpenTelemetry Collectors that will work together to pull in the correct telemetry and apply Kubernetes-specific data to it, which will help correlate issues. Your implementation will also lay the foundation for your applications to send telemetry data, if you choose to instrument them.

Collectors Overview

  1. Each node will contain a Collector, which will use the node’s Kubelet API to gather metrics data about the node and the node’s pod resources.
  2. The entire cluster will contain a separate Collector, which will use the Kubernetes API to get details about Kubernetes Events, such as active deployments.
  3. Applications will be able to use the node’s IP address to send telemetry data (logs, metrics, and traces) to the Collector that is local to the node, if you instrument them.
  4. Each Collector will send telemetry data directly to Honeycomb over gRPC.

Collected Data 

When you finish, you will have access to:

  • Default metrics provided by Kubelet Stats for nodes and pods, plus the optional uptime, *_request_utilization, and *_limit_utilization metrics
  • Default Kubernetes Cluster metrics
  • All Kubernetes events from the cluster
  • Additional Kubernetes metadata, plus all the pod’s labels and annotations as resource attributes whenever possible, including:
    • k8s.namespace.name
    • k8s.deployment.name
    • k8s.statefulset.name
    • k8s.daemonset.name
    • k8s.cronjob.name
    • k8s.job.name
    • k8s.node.name
    • k8s.pod.name
    • k8s.pod.uid
    • k8s.pod.start_time

Before You Begin 

Before beginning this guide, you should have:

  • Created a running Kubernetes cluster.
  • Installed the kubectl command-line utility locally.
  • Installed Helm 3.9+ locally.
  • Deployed some applications to Kubernetes.

You’ll also need your Honeycomb API Key. You can find your Honeycomb API Key in your Environment Settings.

Let’s get started!

Step 1: Create a Namespace 

To help you manage your objects in the cluster, create a namespace to contain the collector infrastructure. In this example, we call the namespace honeycomb.

kubectl create namespace honeycomb

Step 2: Configure Kubernetes with Your Honeycomb API Key 

Within your new namespace, create a Kubernetes Secret that contains your Honeycomb API Key. You can find your Honeycomb API Key in your environment in Honeycomb.

export HONEYCOMB_API_KEY=mykey
kubectl create secret generic honeycomb --from-literal=api-key=$HONEYCOMB_API_KEY --namespace=honeycomb

Step 3: Add OpenTelemetry’s Helm Repository 

OpenTelemetry’s Helm GitHub repository includes Helm charts with all of the resources you need to deploy Collectors to your Kubernetes cluster.

  1. Add the repo:

    helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

  2. Update your repos to ensure Helm is aware of the latest versions:

    helm repo update

Step 4: Deploy Collectors 

Deploy your Collectors–a Deployment-mode Collector to collect the metrics for your cluster and a DaemonSet-mode Collector to collect the application telemetry data and metrics from your cluster’s node(s). You can deploy both Collectors using the same Helm chart, but with different names and values files.

  1. Deploy the Deployment-mode Collector:

    helm install otel-collector-cluster open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/send-data/kubernetes/values-files/values-deployment.yaml

    To see the configuration for the Deployment-mode Collector, download the values-deployment.yaml file.

  2. Deploy the DaemonSet-mode Collector:

    helm install otel-collector open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/send-data/kubernetes/values-files/values-daemonset.yaml

    To see the configuration for the DaemonSet-mode Collector, download the values-daemonset.yaml file.

Warning

If Collector installation fails and returns an error like the following:

Error: INSTALLATION FAILED: template: opentelemetry-collector/templates/service.yaml:38:28: executing "opentelemetry-collector/templates/service.yaml" at <include "opentelemetry-collector.service
InternalTrafficPolicy" .>: error calling include: template: opentelemetry-collector/templates/_helpers.tpl:148:44: executing "opentelemetry-collector.serviceInternalTrafficPolicy" at <eq .Values.
service.enabled true>: error calling eq: incompatible types for comparison

Make sure you have Helm 3.9+ installed, as older Helm versions typically cause this error.

Step 5: Verify the Collector Installation 

Check that the Collectors are installed by using the kubectl command to see if the pods are running:

kubectl get pods --namespace honeycomb

This command should return something like:

NAME                                                             READY   STATUS    RESTARTS   AGE
otel-collector-cluster-opentelemetry-collector-7c9cc9f8d-k9ncw   1/1     Running   0          9m21s
otel-collector-opentelemetry-collector-agent-fcn5v               1/1     Running   0          17m
Note
The result should contain one pod running under a name containing the prefix otel-collector-cluster, and one pod running under a name containing the word agent for each node in your Kubernetes cluster.

You should now have an OpenTelemetry installation in your cluster that can:

  • Receive tracing data from service applications in your cluster and forward it to Honeycomb.
  • Gather and send metrics data from all of the pods in your cluster.
  • Gather and send metrics data about the nodes in your cluster.

Explore Your Data in Honeycomb 

After a few minutes, data should start flowing into Honeycomb.

Tip
If you do not see any data after several minutes, reach out for help in our Pollinators Community Slack. Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

To explore metrics related to your Kubernetes cluster, log in to Honeycomb and query the k8s-metrics dataset in your environment. Try asking the Query Assistant questions like:

  • “Show me the average CPU of my pods”
  • “What’s the P99 memory usage of my nodes?”

To explore the events emitted by Kubernetes itself, query the k8s-events dataset. These may be a little harder to understand, so try asking the Query Assistant questions like:

  • “Show me the pods that have a reason of Started”
  • “Show me pods that are crashing”

For each question you ask, the Query Assistant will create a general query. You can further customize each query by adding visualizations, or by filtering or grouping the data.

What’s Next? 

Now that you have created an observability pipeline and have gotten some metrics, you can use these to get even more visibility into your Kubernetes cluster.

  • Configure your applications to send data to the OpenTelemetry Collectors Have you already instrumented your applications with OpenTelemetry? You’ll need to configure your pods and applications to send data to your new Collectors.

  • Add low-code, automatic instrumentation to your applications Do you want more insight into your application data, but can’t fully instrument your code yet? You can get even more insight by using the OpenTelemetry Operator to automatically instrument your applications. To learn more, visit Low-Code Auto-Instrumentation with the OpenTelemetry Operator for Kubernetes.

  • Get no-code insights into Kubernetes network telemetry Are you working in a coding language that is currently unsupported by OpenTelemetry trace instrumentation? Are you not able to touch your code? Honeycomb can still give you visibility into the traffic traveling between your pods. To learn more about the quickest approach to getting insights into the interconnectivity of your services, visit No-code Kubernetes Insights with the Honeycomb Network Agent.

Additional Information 

The DaemonSet-mode Collector uses the following components:

The Deployment-mode Collector uses the following components:

On this page: