Set Up Honeycomb Refinery

Refinery is designed to sit within your infrastructure where all sources of Honeycomb events can reach it. A standard deployment will have a cluster of servers running Refinery accessible via a load balancer. Refinery instances must be able to communicate with each other to concentrate traces on single servers.

For the quickest way to get started using Refinery:

  1. Meet system requirements before starting
  2. Download and install the latest version of Refinery
  3. Set up your cluster of Refinery processes
  4. Customize your configuration and sampling rules

System Requirements 

To begin, your Refinery cluster requires a minimum of:

  • a linux/amd64 or linux/arm64 operating system
  • 2GB RAM for each server used
  • Access to 2 cores for each server used

In many cases, Refinery only needs one node. If experiencing a large volume of traffic, you may need to scale out to multiple nodes, and likely need a small Redis instance to handle scaling.

We recommend increasing the amount of RAM and the number of cores after your initial set-up. Use our scale and troubleshooting documentation to learn more.

Getting Started 

The recommended way to run Refinery on Kubernetes is with its Helm chart.

Add the Honeycomb Helm repository.

helm repo add honeycomb https://honeycombio.github.io/helm-charts

Update the Honeycomb repository to ensure helm is aware of the latest versions.

helm repo update refinery honeycomb/refinery

Install the latest version of the Refinery Helm chart with default values.

helm install refinery honeycomb/refinery

To install a specific version or override the default values, refer to Refinery Helm chart documentation to learn more about configuration.

Note
Running on containers? We have a Docker image available on Docker Hub.

Find our latest release of Refinery for your operating system and architecture on GitHub.

Use the command line script below to download the latest released .rpm asset for x86_64 from GitHub, install, and then run Refinery:

curl -L -O https://github.com/honeycombio/refinery/releases/download/latest/refinery-2.7.0.x86_64.rpm
rpm -ivh refinery-2.7.0.x86_64.rpm
systemctl start refinery.service

In the above example, we use systemctl to run the Refinery service. Logs can be found at /var/log/journal/refinery.service.log.

Command Line Flags 

The Refinery executable has the following command line flags:

-c, --config=
Path to config file (default: /etc/refinery/refinery.yaml)
-r, --rules_config=
Path to rules config file (default: /etc/refinery/rules.yaml)
-v, --version
Print version number and exit
-d, --debug
If enabled, runs debug service (runs on the first open port between localhost:6060 and :6069 by default). Can be used with a debug service, which allows you to use pprof to visualize and analyze profiling data.
-h, --help
Show help message

Set Up Your Refinery Cluster 

Note
Be sure to keep configuration in-sync between the Refinery processes, so that your traces are consistent.

Ensure that your Refinery cluster(s) meet the minimum system requirements.

We recommend that your list of Refinery peers be configured through Redis (see Redis-based peer management). The Redis server can be small since it only maintains the list of peers. Should the Redis service become unavailable, Refinery instances will continue to use their last known peer list for inter-peer communication.

Note
Refinery does not currently support Redis Cluster or the Redis Sentinel protocol.

For services that are currently sending events directly to Honeycomb’s API, update these services’ Honeycomb API Host property to be the URL for your Refinery cluster’s load balancer to start sampling events with Refinery.

Refinery Configuration 

The default configuration at installation will allow you to run Refinery without any other changes. Be sure to run Refinery in Dry Run Mode to verify your configuration before dropping traffic.

To tune Refinery to your needs, Refinery’s configuration requires additional changes in two files:

As you modify, check out our scale and troubleshooting documentation for recommendations and strategies.