Send System Metrics with the OpenTelemetry Collector

Get host metrics on a regular interval and send them to Honeycomb with an OpenTelemetry Collector.

You can deploy an OpenTelemetry Collector as an agent in your host environment to collect and export diagnostic information about the system. For example, you can get samples of CPU, memory, disk, and network utilization so this data can be queried and visualized in Honeycomb.

Follow this guide to learn how to:

Deploy an OpenTelemetry Collector to collect host metrics.
Configure your Collector using our configuration as a starting point.

Install the Collector Agent

Install the latest OpenTelemetry Collector otelcol-contrib binary on your host system. If you have not installed a Collector before or need a refresher, follow the instructions in Install the Collector on the OpenTelemetry documentation.

Download our honeycomb-metrics-config.yaml file and use that to configure your Collector.

Set Environment Variables

Set the following environment variables before running the Collector:

HNY_API_KEY: Your API key. Replace <YOUR_HONEYCOMB_API_KEY> with this value in the example below.
HNY_DATASET: The name of the dataset you want to send these metrics to. Replace <YOUR_HONEYCOMB_DATASET> with this value in the following example.
OTEL_RESOURCE_ATTRIBUTES: These attributes are added to each data point. Replace hellohostmetrics with the name of your service and development with the name of your environment in the following example.

export HNY_API_KEY=<YOUR_HONEYCOMB_API_KEY> \
export HNY_DATASET=<YOUR_HONEYCOMB_DATASET> \
export OTEL_RESOURCE_ATTRIBUTES='service.name=hellohostmetrics,deployment.environment=development'

Run the Collector Agent

After setting your environment variables, run the Collector binary with the following command.

./otelcol_hny_linux_amd64 --config ./honeycomb-metrics-config.yaml

A process should start and your new Collector agent will send host metrics to your dataset every 10 seconds.

Customize Your Agent Configuration

Our host metrics Collector configuration (honeycomb-metrics-config.yaml) keeps your event volume low and gives you as much data as possible in each event. Use it as the basis for your own configuration, making changes to better fit your observability needs and host environment.

Resource Attributes

You can add custom resource attributes to all your data points by changing the OTEL_RESOURCE_ATTRIBUTES environment variable. Attributes are separated by the comma character (,).

Resource Detection Processor

If your host is deployed using Docker, Google Cloud, Google Kubernetes Engine, Amazon EC2, Amazon ECS, Amazon Elastic Beanstalk, Amazon EKS, or Azure, you can use the resource detection processor to automatically add metadata about the host to every metric data point.

Enable the resource detection processor for a host by adding a value from the list below to processors.resourcedetection.detectors:

Docker: docker
Google Cloud: gce
EC2: ec2
ECS: ecs
Azure: azure

processors:
  resourcedetection/docker:
    detectors: [env, docker]
    timeout: 2s
    override: false

Metrics Collection Interval

Change how often host metrics are collected by setting receivers.hostmetrics.collection_interval.

receivers:
  hostmetrics:
    collection_interval: 10s

Host Metrics Receiver on GitHub.

Disable a Metrics Scraper

You can also disable specific scrapers by removing them from the list in receivers.hostmetrics.scrapers.

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      # disk:

Host Metrics Receiver on GitHub.

Debug Output Verbosity

Change debug logging verbosity by setting exporters.debug.verbosity to either detailed, normal, or basic.

exporters:
  debug:
    verbosity: detailed

Debug Exporter on Github.

Using a Custom Build of OpenTelemetry Collector

It is possible to use a different build of OpenTelemetry Collector to send host metrics to Honeycomb. If you do this, we still recommend using our honeycomb-metrics-config.yaml file as a basis for your configuration, since it is designed to keep your event volume low while providing as much data as possible in each event.

Refer more about managing your metrics data volume, and about using your own Collector with this custom configuration.