1. Docs
2. Refinery
3. Refinery Configuration

Refinery Configuration

Update the fields in config.toml to customize your configuration. The default configuration at installation contains the minimum configuration needed to run Refinery.

Supported sampling methods and their configuration are set in rules.toml.

Default Configuration 

The default Refinery configuration uses a hardcoded peer list for file-based peer management. It uses the DeterministicSampler Sampling Method and a SampleRate of 1, meaning that no traffic will be dropped.

To see the full set of default fields, see GitHub for the full configuration file.

See GitHub for an example configuration file.

General Configuration 

Use the fields below to customize your configuration file.

ListenAddr

The IP and port on which to listen for incoming events. Incoming traffic is expected to be HTTP. If using SSL, put something like nginx in front to do the decryption. Should be in 0.0.0.0:8080 form. HTTP endpoints support both Honeycomb JSON and OpenTelemetry OTLP binary formatted data.

GRPCListenAddr

OPTIONAL The IP and port on which to listen for incoming events over gRPC. A gRPC server will only be started if a non-empty value is provided. Incoming traffic is expected to be unencrypted. If using SSL, put something like nginx in front to do the decryption. Should be in 0.0.0.0:9090 form. Refinery can be configured to receive OpenTelemetry OTLP traffic over gRPC with GRPCListenAddr. If the environment variable REFINERY_GRPC_LISTEN_ADDRESS is set, REFINERY_GRPC_LISTEN_ADDRESS takes precedence and this GRPCListenAddr value is ignored.

PeerListenAddr

The IP and port on which to listen for traffic being rerouted from a peer. Peer traffic is expected to be HTTP. If using SSL, put something like nginx in front to do the decryption. Must be different from the ListenAddr setting. Should be in 0.0.0.0:8081 form.

CompressPeerCommunication

Determines whether Refinery will compress span data it forwards to peers. If it costs money to transmit data between Refinery instances, such as when instances are spread across AWS availability zones, then you almost certainly want compression enabled to reduce your bill. The option to disable compression is provided as an escape hatch for deployments that value lower CPU utilization over data transfer costs.

APIKeys

A list of Honeycomb API keys that the proxy will accept. This list only applies to events as other Honeycomb API actions will fall through to the upstream API directly. Adding keys here causes events arriving with API keys not in this list to be rejected with an HTTP 401 error. If an API key that is a literal ‘*’ is in the list, all API keys are accepted.

HoneycombAPI

The URL for the upstream Honeycomb API.

SendDelay

A short timer that will be triggered when a trace is complete. Refinery will wait this duration before actually sending the trace. The reason for this short delay is to allow for small network delays or clock jitters to elapse and any final spans to arrive before sending the trace. This supports duration strings with supplied units. Set to 0 for immediate sends.

BatchTimeout

BatchTimeout defines the frequency to send unfulfilled batches. Default is the value of DefaultBatchTimeout in libhoney [100ms]. Eligible for live reload.

TraceTimeout

A long timer that represents the outside boundary of how long to wait before sending an incomplete trace. Normally traces are sent when the root span arrives. Sometimes the root span never arrives (due to crashes, for example), and this timer will send a trace even without having received the root span. If you have particularly long-lived traces, you should increase this timer. This supports duration strings with supplied units. It should be set higher (maybe double?) the longest expected trace. If all of your traces complete in under 10 seconds, 30 is a good value here. If you have traces that can last minutes, it should be raised accordingly. Note that the trace does not have to complete before this timer expires, but the sampling decision will be made at that time. So any spans that contain fields that you want to use to compute the sample rate should arrive before this timer expires. Additional spans that arrive after the timer has expired will be sent or dropped according to the sampling decision made when the timer expired.

MaxBatchSize

The number of events to be included in the batch to send. The default value is 500.

SendTicker

A short timer that determines the duration to use to check for traces to send.

LoggingLevel

The level above which we should log. Debug is very verbose, and should only be used in pre-production environments. “error” is the recommended level. Valid options are “debug”, “info”, “error”, and “panic”.

UpstreamBufferSize and PeerBufferSize

Control how large of an event queue to use when buffering events that will be forwarded to peers or the upstream API.

DebugServiceAddr

Sets the IP and port the debug service will run on. The debug service will only run if the command line flag -d is specified. The debug service runs on the first open port between localhost:6060 and localhost:6069 by default.

AddHostMetadataToTrace

Determines whether or not to add information about the host that Refinery is running on to the spans that it processes. If enabled, host metadata will be added to each span using field names prefixed with meta.refinery. (For example, meta.refinery.local_hostname)

EnvironmentCacheTTL

The amount of time a cache entry will live that associates an API key with an environment name. Cache misses look up the environment name using the HoneycombAPI config value. Default is 1 hour (1h). Not eligible for live reload.

AddRuleReasonToTrace

Causes traces that are sent to Honeycomb to include the field meta.refinery.reason. This field contains text indicating which rule was evaluated that caused the trace to be included. Default is false. Eligible for live reload.

AdditionalErrorFields

A list of span fields to include in the logging errors that happen during ingestion of events (for example, the span too large error). Used to track down misbehaving senders in a large installation. The fields dataset, apihost, and environment are always included. Fields not present in the span do not appear in error log. Default is [“trace.span_id”]. Eligible for live reload.

AddSpanCountToRoot

Adds a new metadata field, meta.span_count, to root spans, which indicates the number of child spans on the trace at the time that the sampling decision was made. This value is available to the rules-based sampler, making it possible to write rules that are dependent upon the number of spans in the trace. Default is false. Eligible for live reload.

CacheOverrunStrategy

Controls the cache management behavior under memory pressure. Setting CacheOverrunStrategy to resize means that when a cache overrun occurs, the cache shrinks and never grows again, which is generally not helpful unless occurring because of a permanent change in traffic patterns. Setting CacheOverrunStrategy to impact means that the items having the most impact on the cache size are ejected from the cache earlier than normal, but the cache is not resized. In both cases, CacheOverrunStrategy only applies if MaxAlloc is nonzero. Default is resize for backwards compatibility but impact is recommended for most installations. Eligible for live reload.

Peer Management 

For proper data distribution, each Refinery process needs to know how to identify and communicate with its peers, the other Refinery processes participating in the cluster. The list of peer identifiers can be referenced dynamically through redis (redis-based peer management, recommended) or set explicitly in a hard-coded list in the config file (file-based peer management).

All of the peer management options are set within the [Peer Management] section of the Refinery config file.

Redis-Based Peer Management 

Configuring Refinery for peer management with Redis requires more configuration information than the default file-based peer management, but is recommended so that as a Refinery cluster scales up with new instances, existing instances learn of their new peers without further intervention.

Refinery needs to know the Redis hostname and port, which can be specified in one of two ways:

1. set the REFINERY_REDIS_HOST environment variable or
2. set the RedisHost field in the config file

Similarly, a password for Redis can be specified:

1. set the REFINERY_REDIS_PASSWORD environment variable or
2. set the RedisPassword field in the config file

To customize Redis-based Peer Management for your environment, the following fields can be set under the [Peer Management] section of config.toml:

Type

Set to redis to use redis for managing the peer registry.

RedisHost

Used to connect to redis for peer cluster membership management. If the environment variable REFINERY_REDIS_HOST is set, REFINERY_REDIS_HOST takes precedence and this RedisHost value is ignored. Not eligible for live reload. The redis host should be a hostname and a port. For example: redis.mydomain.com:6379. The example config file has localhost:6379, which will not work with more than one host.

RedisUsername

RedisUsername is the username used to connect to redis for peer cluster membership management. If the environment variable REFINERY_REDIS_USERNAME is set, REFINERY_REDIS_USERNAME takes precedence and this RedisUsername value is ignored. Not eligible for live reload.

RedisPassword

The password used to connect to redis for peer cluster membership management. If the environment variable REFINERY_REDIS_PASSWORD is set, REFINERY_REDIS_PASSWORD takes precedence and this RedisPassword value is ignored. Not eligible for live reload.

UseTLS

Enables TLS when connecting to redis for peer cluster membership management, and sets the MinVersion to 1.2. Not eligible for live reload.

IdentifierInterfaceName

OPTIONAL The name of the network interface to bind to for peer communications. By default, a Refinery instance will register itself in redis using its local hostname as its identifier for peer communications. In environments where domain name resolution is slow or unreliable, override the reliance on name lookups by specifying the name of the peering network interface in this IdentifierInterfaceName field. Refinery will use the first available unicast address on the given interface as its peering identifier to register in redis. The unicast address will be IPv4 by default or IPv6 if UseIPV6Identifier is set to true.

UseIPV6Identifier

OPTIONAL Set to true if the peering network is IPv6 and IdentifierInterfaceName is set. Refinery will use the first IPv6 unicast address found instead of IPv4.

RedisIdentifier

OPTIONAL Explicitly set the peering identifier with which a Refinery instance will register itself in redis. Overrides any automatic use of local hostname and any unicast addresses determined through the use of IdentifierInterfaceName.

File-Based Peer Management 

File-based peer management is the default behavior. This peer management option is not recommended if you expect to increase your Refinery instances due to the intensive process required to update configuration files.

To use file-based Peer Management, configure the following fields in the [Peer Management] section of config.toml:

Type

Set to file to use the Refinery configuration file to list Refinery peers.

Peers

The list of all servers participating in this proxy cluster. Events will be sharded evenly across all peers based on the Trace ID. Values here should be the base URL used to access the peer, and should include scheme, hostname (or ip address), and port. All servers in the cluster should be in this list, including this host.

Environment Variables 

Refinery supports the following environment variables. Environment variables take precedence over file configuration.

Environment Variable	Configuration Field
REFINERY_GRPC_LISTEN_ADDRESS	GRPCListenAddr
REFINERY_REDIS_HOST	PeerManagement.RedisHost
REFINERY_REDIS_USERNAME	PeerManagement.RedisUsername
REFINERY_REDIS_PASSWORD	PeerManagement.RedisPassword
REFINERY_HONEYCOMB_API_KEY	HoneycombLogger.LoggerAPIKey
REFINERY_HONEYCOMB_METRICS_API_KEY REFINERY_HONEYCOMB_API_KEY	HoneycombMetrics.MetricsAPIKey
REFINERY_QUERY_AUTH_TOKEN	QueryAuthToken

Implementation Choices 

There are a few components of Refinery with multiple implementations; the config file lets you choose your desired implementation. For example, there are two logging implementations: one that uses logrus and sends logs to STDOUT, and a honeycomb implementation that sends the log messages to a Honeycomb dataset instead.

Components with multiple implementations have one top level config item that lets you choose which implementation to use and then a section further down with additional config options for that choice. For example, the Honeycomb logger requires an API key.

Changing implementation choices requires a process restart; these changes will not be picked up by a live configuration reload. (Individual configuration options for a given implementation may be eligible for live reload).

Collector 

Collector describes which collector to use for collecting traces. The only current valid option is InMemCollector. More can be added by adding implementations of the Collector interface. Use the fields below to modify your Collector settings.

CacheCapacity

The cache is used to collect all spans into a trace. Its capacity is configured via the CacheCapacity value. For guidance on how to best configure the CacheCapacity value, please refer to the Scale and Troubleshoot documentation. In addition, a cache remembers the sampling decision for any spans that might come in after the trace has been marked “complete” (either by timing out or seeing the root span); that capacity will be 5x this value. This setting is eligible for live reload; growing the cache capacity with a live config reload is fine. Avoid shrinking it with a live reload (you can, but it may cause temporary odd sampling decisions). If the cache capacity is too low, the collect_cache_buffer_overrun metric will increment. If this indicator occurs, you should increase the CacheCapacity value.

MaxAlloc

An optional field that if set, must be an integer >= 0. 64-bit values are supported. If set to a non-zero value, once per tick (see SendTicker) the collector will compare total allocated bytes to this value. If allocation is too high, cache capacity will be reduced and an error will be logged. Useful values for this setting are generally in the range of 75%-90% of available system memory.

Logger 

Logger describes which logger to use for Refinery logs. Valid options are logrus and honeycomb. Set where log events go in this section. Use honeycomb to send logs to the Honeycomb API. Use logrus to send logs to STDOUT.

Honeycomb Logger 

LoggerHoneycombAPI

The URL for the upstream Honeycomb API. Eligible for live reload.

LoggerAPIKey

The API key to use to send log events to the Honeycomb logging dataset. This is separate from the APIKeys used to authenticate regular traffic. Eligible for live reload.

LoggerDataset

The name of the dataset to which to send Refinery logs. Eligible for live reload.

LoggerSamplerEnabled

Enables a dynamic sampler for log messages. This will sample log messages based on [log level:message] key on a per second throughput basis. Not eligible for live reload.

LoggerSamplerThroughput

The per second throughput for each unique log message, when the logger sampler is enabled. Not eligible for live reload.

Logrus Logger 

There are no configurable options for the logrus logger yet.

Metrics 

Metrics describes which service to use for Refinery metrics. Valid options are prometheus and honeycomb. The prometheus option starts a listener that will reply to a request for /metrics. The honeycomb option will send summary metrics to the Honeycomb dataset you specify.

Honeycomb Metrics 

MetricsHoneycombAPI

The URL for the upstream Honeycomb API. Eligible for live reload.

MetricsAPIKey

The API key used to send metrics events to the Honeycomb metrics dataset. This API key is separate from the APIKeys used to authenticate regular traffic. Eligible for live reload.

MetricsDataset

The name of the dataset to which to send Refinery metrics events. Eligible for live reload.

MetricsReportingInterval

The frequency (in seconds) to send metrics events to Honeycomb. Between 1 and 60 is recommended. The default is 1s, but 10s is reasonable. Not eligible for live reload.

Prometheus Metrics 

MetricsListenAddr

Determines the interface and port on which Prometheus will listen for requests for /metrics. Must be different from the main Refinery listener. Not eligible for live reload.

Here ends the list of the general Refinery configuration options. Remember to customize your Sampling Methods configuration to complete your Refinery set-up.