If you are working in a coding language that is currently unsupported by OpenTelemetry trace instrumentation, aren’t able to touch your code (legacy services, anyone?), or just need a fast starting point, Honeycomb can still give you insights into your applications deployed to Kubernetes.
In this guide, you will learn how to get answers about how your applications on Kubernetes are performing in production using the Honeycomb Network Agent, which is a low-effort, no-code, language-agnostic solution to get telemetry for your applications running in Kubernetes. The Network Agent does not require any modification of your application code or application deployments on Kubernetes in order to collect data.
When you finish, you’ll have visibility into application network interactions (for example, source/destination IPs and port numbers), Kubernetes information (for example, source and destination pod names), and application-level, format-specific information (for example, HTTP method and response status code)–all without making any changes to your application code or deployment manifest.
Before beginning this guide, you should have:
honeycomb
.Honeycomb’s Helm GitHub repository includes a Helm chart that contains all of the necessary resources you need to deploy the Honeycomb Network Agent to your Kubernetes cluster.
Add the repo:
helm repo add honeycomb https://honeycombio.github.io/helm-charts
Update your repos to ensure Helm is aware of the latest versions:
helm repo update
Install the Honeycomb Network Agent using the Network Agent Helm chart:
helm install network-agent honeycomb/network-agent --namespace honeycomb
Verify that Helm properly deployed the network agent by using the kubectl
command to see if the pods are running:
kubectl get pod -n honeycomb -o wide
This command should return something like:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hny-network-agent-gn7rw 1/1 Running 0 23s 192.168.42.207 ip-192-168-42-207.us-east-2.compute.internal <none> <none>
hny-network-agent-jw766 1/1 Running 0 23s 192.168.43.44 ip-192-168-43-44.us-east-2.compute.internal <none> <none>
agent
for each node in your Kubernetes cluster.Once you’ve verified that the agents are up and running, you can use kubectl
to look at the startup logs to make sure there are no errors or other issues:
kubectl logs -n honeycomb daemonsets/hny-network-agent
This command should return something like:
{"level":"info","agent_version":"0.2.0-beta","api_key":"******************ABcD","endpoint":"https://api.honeycomb.io","dataset":"hny-network-agent","stats_dataset":"hny-network-agent-stats","time":"2024-01-03T19:13:58Z","message":"Starting Honeycomb Network Agent"}
{"level":"info","time":"2024-01-03T19:13:58Z","message":"Agent is ready!"}
After a few minutes, data should start flowing to Honeycomb.
To see your data, log in to Honeycomb and query the hny-network-agent
dataset in your environment.
Try asking the Query Assistant questions like:
The Network Agent uses a combination of OpenTelemetry Semantic Conventions as of v1.20.0, along with some custom attributes, some meta attributes, and the ability to set arbitrary attributes as needed.
The name
of the event is appropriate for the type of network communication observed.
For example, HTTP events will be named according to the request’s HTTP method.
The duration_ms
of the event is the duration in milliseconds between the beginning of the observed network communication and the beginning of the observed response.
The HTTP attributes mostly follow OpenTelemetry Semantic Convention for HTTP spans.
Some HTTP attributes represent the same attribute but are currently duplicated for convenience of working with multiple semantic conventions.
Field | Description | Example value(s) |
---|---|---|
client.socket.address |
IP address of client | 10.0.1.6 |
http.request.body.size |
Size of request payload body in bytes | 0 ,3495 |
http.request_content_length |
Size of request payload body in bytes | 0 ,3495 |
http.request.header.<header_name> |
The attribute key prefix under which HTTP request headers selected for capture will appear. Header names will appear lowercased with hyphens converted to underscore; for example, a header named X-Custom-Header would appear as an attribute named http.request.header.x_custom_header . User-Agent is captured by default. |
|
http.request.header.user_agent |
HTTP User-Agent header sent by the client. | curl/8.4.0 , CERN-LineMode/2.15 libwww/2.17b3 |
http.request.method |
HTTP request method | GET , POST |
http.method |
HTTP request method | GET , POST |
http.request.timestamp |
The Timestamp the request was captured | 2023-10-18T15:11:45.43095Z |
http.response.body.size |
The size of response payload body in bytes | 25 |
http.response_content_length |
The size of response payload body in bytes | 25 |
http.response.header.<header_name> |
The attribute key prefix under which HTTP response headers selected for capture will appear. Header names will appear lowercased with hyphens converted to underscore; for example, a header named Content-Type would appear as an attribute named http.response.header.content_type . None are captured by default. |
|
http.response.status_code |
HTTP response status code | 200 , 404 , 500 |
http.status_code |
HTTP response status code | 200 , 404 , 500 |
http.response.timestamp |
Timestamp of the response | 2023-10-18T15:11:45.438407Z |
server.socket.address |
IP address of server | 10.0.1.100 |
url.path |
The URI path component | /greeting , /webshop/articles/4 |
http.target |
The URI path component | /greeting , /webshop/articles/4 |
The Kubernetes attributes mostly follow OpenTelemetry Semantic Convention for Kubernetes, with a notable exception of prefixing with source
and destination
.
Field | Description | Example value(s) |
---|---|---|
destination.k8s.container.name |
Name of destination container | frontend |
destination.k8s.namespace.name |
Name of destination namespace | greetings |
destination.k8s.node.name |
Name of destination node | ip-10-0-1-152.ec2.internal |
destination.k8s.node.uid |
UID of destination node | fe5ed02c-4cb6-45f8-ad8b-062287cf3716 |
destination.k8s.pod.name |
Name of destination pod | frontend-go-7db6448ffc-v4bjk |
destination.k8s.pod.uid |
UID of destination pod | 73dbd41e-5ec8-4d88-bd35-25e744285f3e |
destination.k8s.resource.type |
Type of destination resource | pod |
destination.k8s.service.name |
Name of destination service | frontend |
destination.k8s.service.uid |
UID of destination service | 8b0c51a0-e3a8-4958-867a-ecc3edc08f78 |
source.k8s.container.name |
Name of source container | alpine |
source.k8s.namespace.name |
Name of source namespace | default |
source.k8s.node.name |
Name of source node | ip-10-0-1-152.ec2.internal |
source.k8s.node.uid |
UID of source node | fe5ed02c-4cb6-45f8-ad8b-062287cf3716 |
source.k8s.pod.name |
Name of source pod | alpine |
source.k8s.pod.uid |
UID of source pod | 7d8d77bd-c814-428a-8010-93bd8fabbe6f |
source.k8s.resource.type |
Type of source resource | pod |
Field | Description | Example value(s) |
---|---|---|
honeycomb.agent.name |
Name of Network Agent in use | Honeycomb Network Agent |
honeycomb.agent.version |
Version of Network Agent in use | 0.2.0-beta |
meta.agent.node.ip |
IP address of network agent node | 10.0.1.152 |
meta.agent.node.name |
Name of network agent node | ip-10-0-1-152.ec2.internal |
meta.agent.pod.ip |
IP address of network agent pod | 10.0.1.152 |
meta.agent.pod.name |
Name of network agent pod | hny-network-agent-bs7qm |
meta.agent.serviceaccount.name |
Name of network agent service account | honeycomb-sa |
meta.event_handled_at |
Timestamp when event was handled | 2023-10-18T15:11:57.54394747Z |
meta.request.capture_to_handle.latency_ms |
Latency in milliseconds between request event capture and handling | 12112 |
meta.request.packet_count |
Number of TCP packets in the assembled HTTP request | 0 |
meta.response.capture_to_handle.latency_ms |
Latency in milliseconds between response event capture and handling | 12105 |
meta.response.packet_count |
Number of TCP packets in the assembled HTTP response | 1 |
meta.seqack |
Identifier representing corresponding TCP SEQ and ACK numbers | 1765038032 |
meta.stream.ident |
Identifier representing client and server IP address and ports | 10.0.1.6->10.0.1.100:54406->7777:440 |
Error attributes if there is a 4xx
or a 5xx
response status code:
Field | Description | Example value(s) |
---|---|---|
error |
Error field added if there is a 4xx or a 5xx response status code |
HTTP server error ,HTTP client error |
Additional attributes added using OTEL_RESOURCE_ATTRIBUTES
environment variable:
Field | Example value(s) |
---|---|
environment |
dev |
region |
us-east-1 |
Have questions? Encountering difficulties? Missing features? Join our Pollinators Community Slack to ask questions and learn more.