Send Data with the Honeycomb Network Agent (No Code)

If you are working in a coding language that is currently unsupported by OpenTelemetry trace instrumentation, aren’t able to touch your code (legacy services, anyone?), or just need a fast starting point, Honeycomb can still give you insights into your applications deployed to Kubernetes.

In this guide, you will learn how to get answers about how your applications on Kubernetes are performing in production using the Honeycomb Network Agent, which is a low-effort, no-code, language-agnostic solution to get telemetry for your applications running in Kubernetes. The Network Agent does not require any modification of your application code or application deployments on Kubernetes in order to collect data.

When you finish, you’ll have visibility into application network interactions (for example, source/destination IPs and port numbers), Kubernetes information (for example, source and destination pod names), and application-level, format-specific information (for example, HTTP method and response status code)–all without making any changes to your application code or deployment manifest.

Before You Begin 

Before beginning this guide, you should have:

  • Created a running Kubernetes cluster.
  • Created a namespace named honeycomb.

Step 1: Add Honeycomb’s Helm Repository 

Honeycomb’s Helm GitHub repository includes a Helm chart that contains all of the necessary resources you need to deploy the Honeycomb Network Agent to your Kubernetes cluster.

  1. Add the repo:

    helm repo add honeycomb https://honeycombio.github.io/helm-charts
    
  2. Update your repos to ensure Helm is aware of the latest versions:

    helm repo update
    

Step 2: Install the Honeycomb Network Agent 

Install the Honeycomb Network Agent using the Network Agent Helm chart:

helm install network-agent honeycomb/network-agent --namespace honeycomb
Note
This installation will use the default configuration options for the Helm chart. If you want to customize the chart to use your preferred configuration, you can download the chart’s values file.

Step 3: Verify the Network Agent Installation 

Verify that Helm properly deployed the network agent by using the kubectl command to see if the pods are running:

kubectl get pod -n honeycomb -o wide

This command should return something like:

NAME                      READY   STATUS    RESTARTS   AGE   IP               NODE                                           NOMINATED NODE   READINESS GATES
hny-network-agent-gn7rw   1/1     Running   0          23s   192.168.42.207   ip-192-168-42-207.us-east-2.compute.internal   <none>           <none>
hny-network-agent-jw766   1/1     Running   0          23s   192.168.43.44    ip-192-168-43-44.us-east-2.compute.internal    <none>           <none>
Note
Because the Network Agent runs as a DaemonSet, the result should contain one pod running under a name containing the word agent for each node in your Kubernetes cluster.

Once you’ve verified that the agents are up and running, you can use kubectl to look at the startup logs to make sure there are no errors or other issues:

kubectl logs -n honeycomb daemonsets/hny-network-agent

This command should return something like:

{"level":"info","agent_version":"0.2.0-beta","api_key":"******************ABcD","endpoint":"https://api.honeycomb.io","dataset":"hny-network-agent","stats_dataset":"hny-network-agent-stats","time":"2024-01-03T19:13:58Z","message":"Starting Honeycomb Network Agent"}
{"level":"info","time":"2024-01-03T19:13:58Z","message":"Agent is ready!"}

Explore Your Data in Honeycomb 

After a few minutes, data should start flowing to Honeycomb.

To see your data, log in to Honeycomb and query the hny-network-agent dataset in your environment. Try asking the Query Assistant questions like:

  • “Show me the event count of my pods”
  • “What’s my pod event latency?”
Tip
If you do not see any data after several minutes, reach out for help in our Pollinators Community Slack. Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Understanding Network Agent Fields 

The Network Agent uses a combination of OpenTelemetry Semantic Conventions as of v1.20.0, along with some custom attributes, some meta attributes, and the ability to set arbitrary attributes as needed.

The name of the event is appropriate for the type of network communication observed. For example, HTTP events will be named according to the request’s HTTP method. The duration_ms of the event is the duration in milliseconds between the beginning of the observed network communication and the beginning of the observed response.

HTTP Attributes 

The HTTP attributes mostly follow OpenTelemetry Semantic Convention for HTTP spans.

Some HTTP attributes represent the same attribute but are currently duplicated for convenience of working with multiple semantic conventions.

Field Description Example value(s)
client.socket.address IP address of client 10.0.1.6
http.request.body.size Size of request payload body in bytes 0,3495
http.request_content_length Size of request payload body in bytes 0,3495
http.request.header.<header_name> The attribute key prefix under which HTTP request headers selected for capture will appear. Header names will appear lowercased with hyphens converted to underscore; for example, a header named X-Custom-Header would appear as an attribute named http.request.header.x_custom_header. User-Agent is captured by default.
http.request.header.user_agent HTTP User-Agent header sent by the client. curl/8.4.0, CERN-LineMode/2.15 libwww/2.17b3
http.request.method HTTP request method GET, POST
http.method HTTP request method GET, POST
http.request.timestamp The Timestamp the request was captured 2023-10-18T15:11:45.43095Z
http.response.body.size The size of response payload body in bytes 25
http.response_content_length The size of response payload body in bytes 25
http.response.header.<header_name> The attribute key prefix under which HTTP response headers selected for capture will appear. Header names will appear lowercased with hyphens converted to underscore; for example, a header named Content-Type would appear as an attribute named http.response.header.content_type. None are captured by default.
http.response.status_code HTTP response status code 200, 404, 500
http.status_code HTTP response status code 200, 404, 500
http.response.timestamp Timestamp of the response 2023-10-18T15:11:45.438407Z
server.socket.address IP address of server 10.0.1.100
url.path The URI path component /greeting, /webshop/articles/4
http.target The URI path component /greeting, /webshop/articles/4

Kubernetes Attributes 

The Kubernetes attributes mostly follow OpenTelemetry Semantic Convention for Kubernetes, with a notable exception of prefixing with source and destination.

Field Description Example value(s)
destination.k8s.container.name Name of destination container frontend
destination.k8s.namespace.name Name of destination namespace greetings
destination.k8s.node.name Name of destination node ip-10-0-1-152.ec2.internal
destination.k8s.node.uid UID of destination node fe5ed02c-4cb6-45f8-ad8b-062287cf3716
destination.k8s.pod.name Name of destination pod frontend-go-7db6448ffc-v4bjk
destination.k8s.pod.uid UID of destination pod 73dbd41e-5ec8-4d88-bd35-25e744285f3e
destination.k8s.resource.type Type of destination resource pod
destination.k8s.service.name Name of destination service frontend
destination.k8s.service.uid UID of destination service 8b0c51a0-e3a8-4958-867a-ecc3edc08f78
source.k8s.container.name Name of source container alpine
source.k8s.namespace.name Name of source namespace default
source.k8s.node.name Name of source node ip-10-0-1-152.ec2.internal
source.k8s.node.uid UID of source node fe5ed02c-4cb6-45f8-ad8b-062287cf3716
source.k8s.pod.name Name of source pod alpine
source.k8s.pod.uid UID of source pod 7d8d77bd-c814-428a-8010-93bd8fabbe6f
source.k8s.resource.type Type of source resource pod

Metadata Attributes 

Field Description Example value(s)
honeycomb.agent.name Name of Network Agent in use Honeycomb Network Agent
honeycomb.agent.version Version of Network Agent in use 0.2.0-beta
meta.agent.node.ip IP address of network agent node 10.0.1.152
meta.agent.node.name Name of network agent node ip-10-0-1-152.ec2.internal
meta.agent.pod.ip IP address of network agent pod 10.0.1.152
meta.agent.pod.name Name of network agent pod hny-network-agent-bs7qm
meta.agent.serviceaccount.name Name of network agent service account honeycomb-sa
meta.event_handled_at Timestamp when event was handled 2023-10-18T15:11:57.54394747Z
meta.request.capture_to_handle.latency_ms Latency in milliseconds between request event capture and handling 12112
meta.request.packet_count Number of TCP packets in the assembled HTTP request 0
meta.response.capture_to_handle.latency_ms Latency in milliseconds between response event capture and handling 12105
meta.response.packet_count Number of TCP packets in the assembled HTTP response 1
meta.seqack Identifier representing corresponding TCP SEQ and ACK numbers 1765038032
meta.stream.ident Identifier representing client and server IP address and ports 10.0.1.6->10.0.1.100:54406->7777:440

Error attributes if there is a 4xx or a 5xx response status code:

Field Description Example value(s)
error Error field added if there is a 4xx or a 5xx response status code HTTP server error,HTTP client error

Additional attributes added using OTEL_RESOURCE_ATTRIBUTES environment variable:

Field Example value(s)
environment dev
region us-east-1

Example usage 

network agent example showing services and counts

Getting Help 

Have questions? Encountering difficulties? Missing features? Join our Pollinators Community Slack to ask questions and learn more.