Send Metrics from HashiCorp Nomad

HashiCorp Nomadexternal-link is a scheduler that enables teams to deploy and manage containers and non-containerized applications at scale across on-prem and cloud environments.

Configure Nomad to send metrics to Honeycomb with an OpenTelemetry Collector.

Nomad Metrics 

Nomad’s metric endpoint supports Prometheus-formatted metrics. As with other services which expose such an endpoint, use an OpenTelemetry Collector to scrape this endpoint and get these metrics into Honeycomb.

Refer to Nomad’s documentation for a list of key metricsexternal-link, as well as the full telemetry referenceexternal-link.

Configure Nomad 

Prometheus, allocation, and node runtime metrics are not enabled by default.

Therefore, to enable, set to true to enable:

Also, set the prometheus_retention_timeexternal-link value to at least twice the scrape interval of your Collector.

The HashiCorp documentation also suggests setting disable_hostnameexternal-link to avoid having hostname-prefixed metrics.

A suggested configuration can be created as /etc/nomad.d/metrics.hcl within the Nomad client and server, as follows:

Copy
telemetry {
  collection_interval        = "1s"
  disable_hostname           = true
  prometheus_retention_time  = "24h"
  publish_allocation_metrics = true
  publish_node_metrics       = true
  prometheus_metrics         = true
}

Configure the OpenTelemetry Collector 

Scraping the Nomad agent’s Prometheus metrics endpoint requires configuring a OpenTelemetry Collector with a pipeline that starts with a prometheus receiverexternal-link and ends with an OTLP exporterexternal-link. Depending on your chosen method of Nomad deployment, the resource detection processorexternal-link may be helpful to further enrich the OTLP Metrics being sent to Honeycomb.

The following example OpenTelemetry Collector configuration uses the system resource detector processor:

Copy
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: nomad
          scrape_interval: 60s
          metrics_path: /v1/metrics
          params:
            format: ['prometheus']
          static_configs:
            - targets:
              - localhost:4646

processors:
  batch:
  resourcedetection/os:
    detectors:
      - system
    system:
      hostname_sources:
        - os

exporters:
  otlp/metrics:
    endpoint: api.honeycomb.io:443 # US instance
    #endpoint: api.eu1.honeycomb.io:443 # EU instance
    headers:
      "x-honeycomb-team": "YOUR_API_KEY"
      "x-honeycomb-dataset": "nomad-metrics"

service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - resourcedetection/os
        - batch
      exporters:
        - otlp/metrics