OpenTelemetry Collector can be deployed as an agent running on individual hosts, where it will regularly collect and forward diagnostic information about the system it runs on. For example, it can collect regular samples of CPU, memory, disk, and network utilization so that this data can be visualized in a Honeycomb query. These instructions describe how to install and run OpenTelemetry Collector in this way, and send the data it collects into Honeycomb.
OpenTelemetry Collector can be widely deployed across all your hosts as an agent. It can then be configured to scrape a specific set of system metrics at a regular interval (for example, once every 10 seconds), batch that data, and forward it to Honeycomb for querying and display. Each data point will be tagged with information about the host it was collected from, so that you can calculate aggregations across arbitrary subsets of production hosts.
The easiest way to get started collecting host metrics is by using the latest OpenTelemetry Collector otelcol-contrib binary.
Download a binary that is compatible with your system, along with a honeycomb-metrics-config.yaml file.
Run the binary with the command below.
Substitute xxxxx with your API key, hostmetrics with the name of the dataset you would like to send metrics to, otelcol_hny_linux_amd64 with the path to the binary you are using, foozle with your service name, and production with your environment:
HNY_API_KEY=xxxxx \
HNY_DATASET=hostmetrics \
OTEL_RESOURCE_ATTRIBUTES='service.name=foozle,deployment.environment=production' \
./otelcol_hny_linux_amd64 --config ./honeycomb-metrics-config.yaml
A process will start and begin sending metrics to your dataset every 10 seconds.
You can add custom resource attributes to all your data points by changing the OTEL_RESOURCE_ATTRIBUTES environment variable.
Attributes are separated by the comma character (,).
(See more information about attributes and this environment variable.)
If your host is deployed using Docker, Google Cloud, Google Kubernetes Engine, Amazon EC2, Amazon ECS, Amazon Elastic Beanstalk, Amazon EKS, or Azure, you can utilize the resource detection processor to automatically add metadata about the host to every metric data point.
(See more information about the resource detection processor.)
To enable the resource detection processor, edit the configuration file and add a value from the list below to processors.resourcedetection.detectors:
dockergceec2ecsazureYou can change the log level by editing the config file and adjusting exporters.logging.logLevel to debug, info, warn, or error.
See more information about the logging exporter.
You can tweak the collection interval of your hostmetrics by adjusting receivers.hostmetrics.collection_interval.
You can also disable specific scrapers by removing them from the list in receivers.hostmetrics.scrapers.
See more information about the hostmetrics receiver.
It is possible to use a different build of OpenTelemetry Collector to send host metrics to Honeycomb.
If you do this, we still recommend using our honeycomb-metrics-config.yaml file as a basis for your configuration, since it is designed to keep your event volume low while providing you with as much data as possible in each event.
See more about managing your metrics data volume, and about using your own collector with this custom configuration.