HashiCorp Nomad is a scheduler that enables teams to deploy and manage containers and non-containerized applications at scale across on-prem and cloud environments.
Configure Nomad to send metrics to Honeycomb with an OpenTelemetry Collector.
Nomad’s metric endpoint supports Prometheus-formatted metrics. As with other services which expose such an endpoint, use an OpenTelemetry Collector to scrape this endpoint and get these metrics into Honeycomb.
Refer to Nomad’s documentation for a list of key metrics, as well as the full telemetry reference.
Prometheus, allocation, and node runtime metrics are not enabled by default.
Therefore, to enable, set to true to enable:
Also, set the prometheus_retention_time value to at least twice the scrape interval of your Collector.
The HashiCorp documentation also suggests setting disable_hostname to avoid having hostname-prefixed metrics.
A suggested configuration can be created as /etc/nomad.d/metrics.hcl within the Nomad client and server, as follows:
telemetry {
collection_interval = "1s"
disable_hostname = true
prometheus_retention_time = "24h"
publish_allocation_metrics = true
publish_node_metrics = true
prometheus_metrics = true
}
Scraping the Nomad agent’s Prometheus metrics endpoint requires configuring a OpenTelemetry Collector with a pipeline that starts with a prometheus receiver and ends with an OTLP exporter. Depending on your chosen method of Nomad deployment, the resource detection processor may be helpful to further enrich the OTLP Metrics being sent to Honeycomb.
The following example OpenTelemetry Collector configuration uses the system resource detector processor:
receivers:
prometheus:
config:
scrape_configs:
- job_name: nomad
scrape_interval: 60s
metrics_path: /v1/metrics
params:
format: ['prometheus']
static_configs:
- targets:
- localhost:4646
processors:
batch:
resourcedetection/os:
detectors:
- system
system:
hostname_sources:
- os
exporters:
otlp/metrics:
endpoint: api.honeycomb.io:443 # US instance
#endpoint: api.eu1.honeycomb.io:443 # EU instance
headers:
"x-honeycomb-team": "YOUR_API_KEY"
"x-honeycomb-dataset": "nomad-metrics"
service:
pipelines:
metrics:
receivers:
- prometheus
processors:
- resourcedetection/os
- batch
exporters:
- otlp/metrics