1. Docs
2. Investigate
3. Investigate Kubernetes Data

Investigate Kubernetes Data in Honeycomb

Table of Contents

Once your Kubernetes data is in Honeycomb, you can use it to analyze the performance of your Kubernetes applications in production. For example, you can trace application issues to infrastructure causes or pinpoint the users affected by an identified infrastructure issue.

This guide will walk you through the steps required to answer questions like:

• How do resource limits compare to container resource use?
• How does application performance vary with container resource limits?
• Are application errors happening on specific nodes or across the fleet?

Before You Begin 

Before beginning this guide, you should have:

• Created a running Kubernetes cluster.
• Deployed an application to Kubernetes.
• Completed the Kubernetes Quick Start.

Explore Your Data 

In Honeycomb, you can slice and dice your data by Kubernetes attributes from your Home view, or create Boards to save relevant queries and visualizations. Either way, you will want to leverage Honeycomb’s features to create Triggers and detect anomalies using BubbleUp and Correlations.

Slice and Dice Your Data 

Once you have Kubernetes data in Honeycomb, navigate to the Home view and select your Kubernetes dataset to begin exploring your data.

At a minimum, you will see event data, which you can group by various Kubernetes attributes. If you have instrumented your code, you will also see trace data.

Create a Board 

For quick reference over time, you should create a Board that you can customize to show Kubernetes-specific items of interest.

When creating your Board, we recommend that you use one of our customized Board Templates for Kubernetes data, which will get you started with queries and visualizations of particular interest to Kubernetes users. You can locate Board Templates by selecting Explore Templates from the Home view.

Our Kubernetes Board Templates include:

Create Triggers 

Once you have created a Board, you will likely want to configure some Triggers, so you can receive notifications when your data in Honeycomb crosses defined thresholds. Some examples of triggers that may be of interest to Kubernetes users include:

• CPU Use: Pods or nodes that are reaching set CPU limits
• Memory Use: Pods experiencing OOMKilled or nodes that are reaching a certain memory usage limit
• Unhealthy pods: Pods that are experiencing a problematic status in Kubernetes Events, such as a reason of BackOff, Failed, Err, or Unhealthy

To create a Trigger:

1. On your Board, locate a Query from which you want to create a Trigger.
2. Select the query to open it in the Query Builder display.
3. Select the three-dot overflow menu, located to the left of Run Query, and select Make Trigger.
4. Configure the trigger by defining trigger details, an alert threshold, and your notification preferences.

Investigate Anomalies 

Follow our guided Kubernetes sandbox scenario to see how you can simplify debugging by using the core analysis loop–Honeycomb BubbleUp, Correlations, and rich queryable data–to link application behavior to underlying infrastructure.

Examples 

Let’s look at some examples to learn how to use Honeycomb to investigate some common issues.

Trace an Application Issue to Infrastructure 

The OpenTelemetry Kubernetes Attributes Processor adds Kubernetes context to your telemetry, allowing for correlation with an application’s traces, metrics, and logs. With this data now on our spans, let’s investigate some slow traces and identify the cause of them.

Find Slow Traces 

To find slow traces:

1. In Query Builder, enter VISUALIZE HEATMAP(duration_ms).
2. Select Run Query. This creates a heatmap below the Query Builder. The slowest traces appear towards the top of the heatmap.
3. In the heatmap, select a slow request towards the top of the chart. In this example, it appears as a teal square with a high duration.
4. In the menu that appears, select View trace.

The trace’s detailed Trace Waterfall view appears next. In this example, the span is very slow when communicating between the audit and storage services. In addition, a span, displayed in red, contains an error. Select the errored span and use the Trace Sidebar to view the errored span’s fields and attributes. This information can provide some clues, but using BubbleUp would be faster.

Identify the Cause 

To investigate further, return to the previous query and use BubbleUp:

1. In the top left of the Trace Waterfall view, select the back arrow next to “Query in all datasets”. The previous Query Results page with the heatmap appears.
2. In the heatmap, draw a box around the slow trace data to define the selection. A menu appears.
3. Select Detect Anomalies (BubbleUp). The BubbleUp charts appear below the heatmap. BubbleUp creates charts to show differences between the selected slow requests and all other requests returned for the time window.

In our example BubbleUp results, one pod (k8s.pod.name) looks to be a significant outlier and that audit job (audit_id) 130720 is failing.

Find Correlations 

Now, let’s see if there are any correlations between our previously identified application issue and our infrastructure. Within the query results, select the Correlations tab below the heatmap. The dropdown windows allows us to use a pre-existing Board with saved queries to correlate data with our Query Results.

In our example, our Correlations board is Kubernetes Pod Metrics, which is available as a Board Template for your own use. The Correlations results show two indicators: Spikes in pod memory, and CPU consumption.

Hover over the spikes in the Kubernetes Pod Start Events Correlation chart. A line appears on all charts in the display to indicate the same point in time. The hovering reveals that the Storage pod spikes at the same time as the slow requests on the heatmap. Each spike has a Started event, as seen in the Kubernetes Pod Start Events chart, the last chart on the right. This means that Kubernetes is restarting the Storage service’s container.

We can conclude that the Storage service does not have enough resources to process and store audit job 130720, which leads to the application issues we originally noticed.

Pinpoint Users Affected by an Infrastructure Issue 

You can use the Kubernetes Workload Health Board to monitor and investigate infrastructure issues, and in conjunction with Query Builder, identify affected users. (Kubernetes Workload Health Board is available as a Board Template, which you can use and apply to your data.)

In our example, the Unhealthy Nodes Query in our Kubernetes Workload Health Board is showing unhealthy nodes appearing off and on between the 23rd and 27th of October.

To investigate, go to Query Builder to see what the application performance looks like for users during this time range. In Query Builder:

1. Run a query with:

   VISUALIZE	WHERE	GROUP BY
   HEATMAP(duration_ms)	user.id exists	user.id

2. Use the time picker to adjust the time window to the Last 7 days.

3. Select Run Query.

In our example’s query results, it looks like there are occasional spikes in slow requests, similar to the spikes in unhealthy nodes previously noticed on the Kubernetes Workload Health board.

To confirm this:

1. Select the Correlations tab below the heatmap.
2. Select the dropdown window that displays the selected data source.
3. Choose the Kubernetes Workload Health board from the available options.

In our example’s Correlations display, the slow requests from our query and spikes in unhealthy nodes from the Unhealthy Nodes chart align. Clumps of pending pods are also seen in the Pending Pods chart at this same time too.

To learn more about the users affected:

• Select the Overview tab to the left of the Correlations tab. The query being used includes GROUP BY user.id, which allows you to group results by that field and see them listed in the Overview tab’s summary table. Hovering over each user.id’s row adjusts the above heatmap display of slow traces, so another correlation between a specific affected user and slow traces can be determined.

• Try adding additional fields to the existing query, such as user information like user.email to the GROUP BY clause. The goal is to give more readable and potentially actionable information about the affected users.

Identify Traffic Source 

The Honeycomb Network Agent captures raw network packets from the network interface that is shared by all resources on a Kubernetes node, assembles them into whole payloads, parses them, and then converts them to events.

In conjunction with the Network Agent, you can use Derived Columns to identify the vector of network traffic and differentiate between internal cluster traffic and application traffic.

Create a Derived Column that Analyzes Traffic Vectors 

Network traffic events contain source.* and destination.* fields, which you can use to create a Derived Column that identifies whether traffic came from outside your cluster and whether it crossed namespaces.

Create a Derived Column and enter the display name traffic_vector and the following Function:

IF(NOT(EXISTS($source.k8s.node.name)), "inbound",
    IF(NOT(EXISTS($destination.k8s.node.name)), "outbound",
        IF(EQUALS($destination.k8s.namespace.name, $source.k8s.namespace.name), "cross-pod", "cross-namespace")
    )
)

Create a Derived Column that Differentiates Between Systems 

Inside a Kubernetes cluster, many HTTP requests perform various tasks, such as completing health checks and interacting with the cloud. You can create a Derived Column that filters traffic inside the cluster out by examining the user agent.

Create a Derived Column and enter the display name is_system_traffic and the following Function:

IF(STARTS_WITH($user_agent.original, "kube-probe"), true,
    IF(STARTS_WITH(differe$user_agent.original, "aws-sdk-go"), true,
        IF(STARTS_WITH($user_agent.original, "amazon-vpc-cni-k8s"), true, false)
    )
)

Create a Query Using Your Derived Columns 

Use your new Derived Columns (traffic_vector and is_system_traffic) to create a Query that filters out system traffic and shows application traffic with vector.

In Query Builder, run a query with:

In our example’s query results, specific details appears about the network traffic.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit