If you are running applications that are not using OpenTelemetry in your Kubernetes cluster, you can still collect the logs from your containers.
In this guide, you will learn how to use an OpenTelemetry Collector to get additional insight into your data by sending your container logs to Honeycomb using OpenTelemetry’s Filelog Receiver.
Before beginning this guide, you should have:
Enable the Filelog Receiver to collect logs by adding the logsCollection
preset in the values file for your OpenTelemetry DaemonSet-mode Collector. Place it near the top of the values file under the config
section:
presets:
logsCollection:
enabled: true
mode: daemonset
image:
repository: otel/opentelemetry-collector-k8s
# Required to use the kubeletstats cpu/memory utilization metrics
clusterRole:
create: true
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
verbs:
- get
extraEnvs:
- name: HONEYCOMB_API_KEY
valueFrom:
secretKeyRef:
name: honeycomb
key: api-key
presets:
# enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
kubernetesAttributes:
enabled: true
extractAllPodLabels: true
extractAllPodAnnotations: true
# enables the kubeletstatsreceiver and adds it to the metrics pipelines
kubeletMetrics:
enabled: true
config:
receivers:
jaeger: null
zipkin: null
kubeletstats:
insecure_skip_verify: true # required as most clusters use self-signed certificates
collection_interval: 30s
metric_groups:
- node
- pod
metrics:
k8s.node.uptime:
enabled: true
k8s.pod.uptime:
enabled: true
k8s.pod.cpu_limit_utilization:
enabled: true
k8s.pod.cpu_request_utilization:
enabled: true
k8s.pod.memory_limit_utilization:
enabled: true
k8s.pod.memory_request_utilization:
enabled: true
exporters:
otlp:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "${env:HONEYCOMB_API_KEY}"
otlp/k8s-metrics:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "${env:HONEYCOMB_API_KEY}"
"x-honeycomb-dataset": "k8s-metrics"
otlp/k8s-logs:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "${env:HONEYCOMB_API_KEY}"
"x-honeycomb-dataset": "k8s-logs"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp]
metrics:
exporters: [otlp/k8s-metrics]
logs:
exporters: [otlp/k8s-logs]
ports:
jaeger-compact:
enabled: false
jaeger-thrift:
enabled: false
jaeger-grpc:
enabled: false
zipkin:
enabled: false
By default, the logsCollection
preset in the OpenTelemetry Helm chart will configure the Collector to collect all the pod logs in a cluster.
In larger clusters, you may want to configure the preset to target specific pods or applications to avoid being overwhelmed.
To collect only logs in a specific directory or with a specific filename, combine the preset with some configuration:
presets:
logsCollection:
enabled: true
config:
receivers:
filelog:
include:
- /var/log/pods/my-namespace*/*/*.log
- /var/log/pods/*mypodname*/*/*.log
- /var/log/pods/*/my-containername/*.log
To collect logs for only certain label selectors, use the Kubernetes Attributes processor to enable labels, and then the Filter processor to remove unwanted data:
presets:
kubernetesAttributes:
enabled: true
extractAllPodLabels: true
logsCollection:
enabled: true
config:
processors:
filter:
error_mode: ignore
logs:
log_record:
- resource.attributes["app.kubernetes.io/component"] != "myapp1"
exporters:
otlp:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "YOUR_API_KEY"
"x-honeycomb-dataset": "myapp1-logs"
service:
pipelines:
logs:
receivers:
- filelog
processors:
- memory_limiter
- k8sattributes
- filter
- batch
exporters:
- otlp
To send specific logs to different datasets, use the Filter processor and multiple OTLP exporters:
presets:
kubernetesAttributes:
enabled: true
extractAllPodLabels: true
logsCollection:
enabled: true
config:
processors:
filter/myapp1:
error_mode: ignore
logs:
log_record:
- resource.attributes["app.kubernetes.io/component"] != "myapp1"
filter/myapp2:
error_mode: ignore
logs:
log_record:
- resource.attributes["app.kubernetes.io/component"] != "myapp2"
exporters:
otlp/myapp1:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "YOUR_API_KEY"
"x-honeycomb-dataset": "myapp1-logs"
otlp/myapp2:
endpoint: "api.honeycomb.io:443" # US instance
#endpoint: "api.eu1.honeycomb.io:443" # EU instance
headers:
"x-honeycomb-team": "YOUR_API_KEY"
"x-honeycomb-dataset": "myapp2-logs"
service:
pipelines:
logs:
receivers:
- filelog
processors:
- memory_limiter
- k8sattributes
- filter/myapp1
- batch
exporters:
- otlp/myapp1
logs/myapp2:
receivers:
- filelog
processors:
- memory_limiter
- k8sattributes
- filter/myapp2
- batch
exporters:
- otlp/myapp2