Once you have a telemetry pipeline in place, you can gather metrics, logs, and traces from your deployed applications by adding automatic instrumentation to your applications. You can do this with very little code by installing the OpenTelemetry Operator for Kubernetes in your cluster. Once installed, the Operator will monitor when pods are created and check whether it should sideload automatic instrumentation packages.
We consider this approach to be “low-code” because it requires you to access and modify the Kubernetes deployment manifests for your application. This approach also pushes automatic instrumentation into your code base as the application starts, so that trace data can be gathered. Although this is a very minimal amount of change overall and does not require you to directly modify any application code, we differentiate this from “no-code” because it does require that you modify how your applications are deployed on Kubernetes.
If you do not have access to change your application’s deployment manifests on your Kubernetes cluster, then take a look at the “no-code” Honeycomb Network Agent, which installs in Kubernetes and investigates traffic by monitoring network transactions between pods.
Supported languages include:
Before beginning this guide, you should have:
honeycomb. Install the OpenTelemetry Operator Pods into your cluster using a Helm chart:
helm install \
--set admissionWebhooks.certManager.enabled=false \
--set admissionWebhooks.autoGenerateCert.enabled=true \
--set manager.collectorImage.repository="otel/opentelemetry-collector-k8s" \
--namespace honeycomb \
--create-namespace \
opentelemetry-operator open-telemetry/opentelemetry-operator
manager.featureGates configuration line to the command.
If you are working with a different programming language, you can omit this line.admissionWebhooks.certManager.enabled to false and admissionWebhooks.autoGenerateCert.enabled to true. We do not recommend that you use this configuration in a production environment, but setting up a cert-manager is outside the scope of this document. Check that the Operator is installed by using the kubectl command to see if the pod is running:
kubectl get pods --namespace honeycomb
This command should return something like:
NAME READY STATUS RESTARTS AGE
opentelemetry-operator-567bc4ff75-p287g 2/2 Running 0 21s
opentelemetry-operator.After verifying that the pod is running correctly, you can monitor the work the operator does by tailing the logs with the kubectl command:
kubectl logs -n honeycomb -f opentelemetry-operator-567bc4ff75-p287g
This command should return something like:
{"level":"info","ts":"2023-09-22T15:21:27Z","msg":"Starting Controller","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"}
{"level":"info","ts":"2023-09-22T15:21:27Z","logger":"instrumentation-resource","msg":"default","name":"otel-autoinstrumentation"}
{"level":"info","ts":"2023-09-22T15:21:27Z","logger":"collector-upgrade","msg":"no instances to upgrade"}
{"level":"info","ts":"2023-09-22T15:21:27Z","logger":"instrumentation-resource","msg":"validate update","name":"otel-autoinstrumentation"}
{"level":"info","ts":"2023-09-22T15:21:27Z","msg":"Starting workers","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","worker count":1}
{"level":"info","ts":"2023-09-22T15:34:24Z","logger":"instrumentation-resource","msg":"default","name":"otel-autoinstrumentation"}
{"level":"info","ts":"2023-09-22T15:34:24Z","logger":"instrumentation-resource","msg":"validate update","name":"otel-autoinstrumentation"}
Deploy an Instrumentation manifest to your Kubernetes cluster, which will enable the OpenTelemetry Operator to automatically instrument your services:
kubectl apply --namespace honeycomb -f https://docs.honeycomb.io/send-data/kubernetes/values-files/otel-autoinstrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: otel-autoinstrumentation
spec:
exporter:
endpoint: http://$(OTEL_NODE_IP):4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env:
# Required if endpoint is set to 4317.
# Python autoinstrumentation uses http/proto by default
# so data must be sent to 4318 instead of 4317.
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://$(OTEL_NODE_IP):4318
dotnet:
env:
# Required if endpoint is set to 4317.
# Dotnet autoinstrumentation uses http/proto by default
# See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/888e2cd216c77d12e56b54ee91dafbc4e7452a52/docs/config.md\#otlp
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://$(OTEL_NODE_IP):4318
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.10.0
env:
# Required if endpoint is set to 4317.
# Java autoinstrumentation agent v2.0+ uses http/proto by default
# See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/9bbfe7fe4e3f65cb698d6d2320ac87372d5d572f/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/OtlpProtocolPropertiesSupplier.java#L19
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://$(OTEL_NODE_IP):4318
nodejs:
env:
# Optional.
# We recommend disabling fs automatic instrumentation because
# it can be noisy and expensive during startup
- name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
value: fs
This manifest instructs the Operator to perform the following actions whenever you add annotations to your pods:
In the deployment manifest for your application, add the required annotations. These will vary according to programming language.
For example, our Go application requires the following annotations. Note where the annotations are located in the manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-go-app
spec:
replicas: 1
template:
metadata:
labels:
app: my-go-app
annotations:
# Added Annotations for auto-instrumentation go here - points at the namespace where the instrumentation
# manifest is installed and the name of meta.name field in the instrumentation manifest
instrumentation.opentelemetry.io/inject-go: "honeycomb/otel-autoinstrumentation"
# Go Requires an additional annotation to the path of the binary on the container
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/my-go-app"
spec:
serviceAccountName: my-go-app
Each supported programming language requires different annotations:
annotations:
instrumentation.opentelemetry.io/inject-dotnet: "honeycomb/otel-autoinstrumentation"
annotations:
instrumentation.opentelemetry.io/inject-java: "honeycomb/otel-autoinstrumentation"
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "honeycomb/otel-autoinstrumentation"
annotations:
instrumentation.opentelemetry.io/inject-python: "honeycomb/otel-autoinstrumentation"
Go requires an extra annotation that tells the Operator where to find the Go binary of your application on the container.
annotations:
instrumentation.opentelemetry.io/inject-go: "honeycomb/otel-autoinstrumentation"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/app/frontend-service"
Possible values for the annotation include:
"true" - inject an Instrumentation Kubernetes Custom Resource (CR) instance from the current namespace"my-instrumentation" - inject a specific Instrumentation CR instance from the current namespace"my-other-namespace/my-instrumentation" - inject a specific Instrumentation CR instance from another namespace"false" - do not inject an Instrumentation CR instance The next time requests are made to your application, data should start flowing to Honeycomb. Once your applications are sending data, you can explore trace data and gain insights from the Kubernetes attributes that have been attached by the Kubernetes Attributes Processor in your Collector.