Refinery Configuration

Update the fields in config.toml to customize your configuration. The default configuration at installation contains the minimum configuration needed to run Refinery.

Supported sampling methods and their configuration are set in rules.toml.

Default Configuration 🔗

The default Refinery configuration uses a hardcoded peer list for file-based peer management. It uses the DeterministicSampler Sampling Method and a SampleRate of 1, meaning that no traffic will be dropped.

To see the full set of default fields, see GitHub for the full configuration file.

See GitHub for an example configuration file.

General Configuration 🔗

Use the fields below to customize your configuration file.

ListenAddr: The IP and port on which to listen for incoming events. Incoming traffic is expected to be HTTP. If using SSL, put something like nginx in front to do the decryption. Should be in 0.0.0.0:8080 form. HTTP endpoints support both Honeycomb JSON and OpenTelemetry OTLP binary formatted data.
GRPCListenAddr: OPTIONAL The IP and port on which to listen for incoming events over gRPC. A gRPC server will only be started if a non-empty value is provided.Incoming traffic is expected to be unencrypted. If using SSL, put something like nginx in front to do the decryption. Should be in 0.0.0.0:9090 form. Refinery can be configured to receive OpenTelemetry OTLP traffic over gRPC with GRPCListenAddr.
PeerListenAddr: The IP and port on which to listen for traffic being rerouted from a peer. Peer traffic is expected to be HTTP. If using SSL, put something like nginx in front to do the decryption. Must be different from the ListenAddr setting. Should be in 0.0.0.0:8081 form.
CompressPeerCommunication: Determines whether Refinery will compress span data it forwards to peers. If it costs money to transmit data between Refinery instances, such as when instances are spread across AWS availability zones, then you almost certainly want compression enabled to reduce your bill. The option to disable compression is provided as an escape hatch for deployments that value lower CPU utilization over data transfer costs.
APIKeys: A list of Honeycomb API keys that the proxy will accept. This list only applies to events as other Honeycomb API actions will fall through to the upstream API directly. Adding keys here causes events arriving with API keys not in this list to be rejected with an HTTP 401 error. If an API key that is a literal ‘*’ is in the list, all API keys are accepted.
HoneycombAPI: The URL for the upstream Honeycomb API.
SendDelay: A short timer that will be triggered when a trace is complete. Refinery will wait this duration before actually sending the trace. The reason for this short delay is to allow for small network delays or clock jitters to elapse and any final spans to arrive before sending the trace. This supports duration strings with supplied units. Set to 0 for immediate sends.
TraceTimeout: A long timer that represents the outside boundary of how long to wait before sending an incomplete trace. Normally traces are sent when the root span arrives. Sometimes the root span never arrives (due to crashes, for example), and this timer will send a trace even without having received the root span. If you have particularly long-lived traces, you should increase this timer. This supports duration strings with supplied units. It should be set higher (maybe double?) the longest expected trace. If all of your traces complete in under 10 seconds, 30 is a good value here. If you have traces that can last minutes, it should be raised accordingly. Note that the trace doesn’t have to complete before this timer expires, but the sampling decision will be made at that time. So any spans that contain fields that you want to use to compute the sample rate should arrive before this timer expires. Additional spans that arrive after the timer has expired will be sent or dropped according to the sampling decision made when the timer expired.
MaxBatchSize: The number of events to be included in the batch to send. The default value is 500.
SendTicker: A short timer that determines the duration to use to check for traces to send.
LoggingLevel: The level above which we should log. Debug is very verbose, and should only be used in pre-production environments. “info” is the recommended level. Valid options are “debug”, “info”, “error”, and “panic”.
UpstreamBufferSize and PeerBufferSize: Control how large of an event queue to use when buffering events that will be forwarded to peers or the upstream API.
DebugServiceAddr: Sets the IP and port the debug service will run on. The debug service will only run if the command line flag -d is specified. The debug service runs on the first open port between localhost:6060 and localhost:6069 by default.
AddHostMetadataToTrace: Determines whether or not to add information about the host that Refinery is running on to the spans that it processes. If enabled, host metadata will be added to each span using field names prefixed with meta.refinery. (For example, meta.refinery.local_hostname)

Peer Management 🔗

For proper data distribution, each Refinery process needs to know how to identify and communicate with its peers, the other Refinery processes participating in the cluster. The list of peer identifiers can be referenced dynamically through redis (redis-based peer management, recommended) or set explicitly in a hard-coded list in the config file (file-based peer management).

All of the peer management options are set within the [Peer Management] section of the Refinery config file.

Redis-Based Peer Management 🔗

Configuring Refinery for peer management with Redis requires more configuration information than the default file-based peer management, but is recommended so that as a Refinery cluster scales up with new instances, existing instances learn of their new peers without further intervention.

Refinery needs to know the Redis hostname and port, which can be specified in one of two ways:

set the REFINERY_REDIS_HOST environment variable or
set the RedisHost field in the config file

Similarly, a password for Redis can be specified:

set the REFINERY_REDIS_PASSWORD environment variable or
set the RedisPassword field in the config file

To customize Redis-based Peer Management for your environment, the following fields can be set under the [Peer Management] section of config.toml:

Type: Set to redis to use redis for managing the peer registry.
RedisHost: Used to connect to redis for peer cluster membership management. If the environment variable REFINERY_REDIS_HOST is set, REFINERY_REDIS_HOST takes precedence and this RedisHost value is ignored. Not eligible for live reload. The redis host should be a hostname and a port. For example: redis.mydomain.com:6379. The example config file has localhost:6379, which will not work with more than one host.
RedisPassword: The password used to connect to redis for peer cluster membership management. If the environment variable REFINERY_REDIS_PASSWORD is set, REFINERY_REDIS_PASSWORD takes precedence and this RedisPassword value is ignored. Not eligible for live reload.
UseTLS: Enables TLS when connecting to redis for peer cluster membership management, and sets the MinVersion to 1.2. Not eligible for live reload.
IdentifierInterfaceName: OPTIONAL The name of the network interface to bind to for peer communications. By default, a Refinery instance will register itself in redis using its local hostname as its identifier for peer communications. In environments where domain name resolution is slow or unreliable, override the reliance on name lookups by specifying the name of the peering network interface in this IdentifierInterfaceName field. Refinery will use the first available unicast address on the given interface as its peering identifier to register in redis. The unicast address will be IPv4 by default or IPv6 if UseIPV6Identifier is set to true.
UseIPV6Identifier: OPTIONAL Set to true if the peering network is IPv6 and IdentifierInterfaceName is set. Refinery will use the first IPv6 unicast address found instead of IPv4.
RedisIdentifier: OPTIONAL Explicitly set the peering identifier with which a Refinery instance will register itself in redis. Overrides any automatic use of local hostname and any unicast addresses determined through the use of IdentifierInterfaceName.

File-Based Peer Management 🔗

File-based peer management is the default behavior. This peer management option is not recommended if you expect to increase your Refinery instances due to the intensive process required to update configuration files.

To use file-based Peer Management, configure the following fields in the [Peer Management] section of config.toml:

Type: Set to file to use the Refinery configuration file to list Refinery peers.
Peers: The list of all servers participating in this proxy cluster. Events will be sharded evenly across all peers based on the Trace ID. Values here should be the base URL used to access the peer, and should include scheme, hostname (or ip address), and port. All servers in the cluster should be in this list, including this host.

Implementation Choices 🔗

There are a few components of Refinery with multiple implementations; the config file lets you choose your desired implementation. For example, there are two logging implementations: one that uses logrus and sends logs to STDOUT, and a honeycomb implementation that sends the log messages to a Honeycomb dataset instead.

Components with multiple implementations have one top level config item that lets you choose which implementation to use and then a section further down with additional config options for that choice. For example, the Honeycomb logger requires an API key.

Changing implementation choices requires a process restart; these changes will not be picked up by a live configuration reload. (Individual configuration options for a given implementation may be eligible for live reload).

Collector 🔗

Collector describes which collector to use for collecting traces. The only current valid option is InMemCollector. More can be added by adding implementations of the Collector interface. Use the fields below to modify your Collector settings.

CacheCapacity: The cache is used to collect all spans into a trace. Its capacity is configured via the CacheCapacity value. For guidance on how to best configure the CacheCapacity value, please refer to the Scale and Troubleshoot documentation. In addition, a cache remembers the sampling decision for any spans that might come in after the trace has been marked “complete” (either by timing out or seeing the root span); that capacity will be 5x this value. This setting is eligible for live reload; growing the cache capacity with a live config reload is fine. Avoid shrinking it with a live reload (you can, but it may cause temporary odd sampling decisions). If the cache capacity is too low, the collect_cache_buffer_overrun metric will increment. If this indicator occurs, you should increase the CacheCapacity value.
MaxAlloc: An optional field that if set, must be an integer >= 0. 64-bit values are supported. If set to a non-zero value, once per tick (see SendTicker) the collector will compare total allocated bytes to this value. If allocation is too high, cache capacity will be reduced and an error will be logged. Useful values for this setting are generally in the range of 75%-90% of available system memory.

Logger 🔗

Logger describes which logger to use for Refinery logs. Valid options are logrus and honeycomb. Set where log events go in this section. Use honeycomb to send logs to the Honeycomb API. Use logrus to send logs to STDOUT.

Honeycomb Logger 🔗

LoggerHoneycombAPI: The URL for the upstream Honeycomb API. Eligible for live reload.
LoggerAPIKey: The API key to use to send log events to the Honeycomb logging dataset. This is separate from the APIKeys used to authenticate regular traffic. Eligible for live reload.
LoggerDataset: The name of the dataset to which to send Refinery logs. Eligible for live reload.
LoggerSamplerEnabled: Enables a dynamic sampler for log messages. This will sample log messages based on [log level:message] key on a per second throughput basis. Not eligible for live reload.
LoggerSamplerThroughput: The per second throughput for each unique log message, when the logger sampler is enabled. Not eligible for live reload.

Logrus Logger 🔗

There are no configurable options for the logrus logger yet.

Metrics 🔗

Metrics describes which service to use for Refinery metrics. Valid options are prometheus and honeycomb. The prometheus option starts a listener that will reply to a request for /metrics. The honeycomb option will send summary metrics to the Honeycomb dataset you specify.

Honeycomb Metrics 🔗

Refinery emits metrics as a Honeycomb event containing all values at each reporting interval. This configuration does not send OTLP Metrics to Honeycomb.

MetricsHoneycombAPI: The URL for the upstream Honeycomb API. Eligible for live reload.
MetricsAPIKey: The API key used to send metrics events to the Honeycomb metrics dataset. This API key is separate from the APIKeys used to authenticate regular traffic. Eligible for live reload.
MetricsDataset: The name of the dataset to which to send Refinery metrics events. Eligible for live reload.
MetricsReportingInterval: The frequency (in seconds) to send metrics events to Honeycomb. Between 1 and 60 is recommended. The default is 1s, but 10s is reasonable. Not eligible for live reload.

Prometheus Metrics 🔗

MetricsListenAddr: Determines the interface and port on which Prometheus will listen for requests for /metrics. Must be different from the main Refinery listener. Not eligible for live reload.

Here ends the list of the general Refinery configuration options. Remember to customize your Sampling Methods configuration to complete your Refinery set-up.

Honeycomb.io Docs