1. Docs
  2. Troubleshoot
  3. Recommended Migrations
  4. Upgrade Refinery

Upgrading Refinery

This guide walks you through upgrading Refinery across major versions to access new features and improvements.

Upgrading to Refinery 3.0 

Refinery 3.0 introduces significant improvements and includes breaking changes that require updating your configuration.

Warning
If you’re upgrading from Refinery 1.x, first upgrade to Refinery 2.x using the Configuration Conversion Tool to convert your file format, then follow these Refinery 3.0 upgrade instructions.

What’s New in Refinery 3.0 

Upgrading your Refinery instance to 3.0 provides:

  • More efficient CPU and memory utilization
  • Enhanced cluster stability and reliability

Update Your Configuration 

Update your Refinery configuration before upgrading to version 3.0. Your cluster will fail to start or behave as intended if you upgrade without updating your configuration first.

Options to Remove 

Refinery 3.0 eliminates several configuration options that are now handled automatically or replaced with new features.

Trace Redistribution 

Refinery 3.0 no longer supports redistributing in-progress traces during scaling events. Remove these options:

  • DisableRedistribution
  • RedistributionDelay
Outgoing Buffer Sizes 

Refinery 3.0 automatically manages upstream and peer queue sizes. Remove these options:

  • PeerBufferSize
  • UpstreamBufferSize
Trace Locality Mode (Experimental) 

Refinery 3.0 removes the Trace Locality experimental feature. Remove these options:

  • TraceLocalityMode
  • MaxDropDecisionBatchSize
  • DropDecisionSendInterval
  • MaxKeptDecisionBatchSize
  • KeptDecisionSendInterval

Options to Replace 

Some Refinery 2.x settings have been renamed or redesigned in version 3.0.

CacheCapacity 

Replace CacheCapacity with queue size options:

  1. Remove CacheCapacity from your configuration.
  2. Add IncomingQueueSize and PeerQueueSize (if they aren’t already present).
  3. Set both queue sizes to 3 times your previous CacheCapacity value. For example, if CacheCapacity was 1000, set both IncomingQueueSize and PeerQueueSize to 3000.
LegacyMetrics 

Replace LegacyMetrics with OTelMetrics:

  1. Replace LegacyMetrics with OTelMetrics in your configuration.
  2. Keep the same configuration options. Both metrics types use identical settings.
Note
Refinery deprecated LegacyMetrics in version 2.0 and removes it in version 3.0. Some metric names will change slightly after this update.

Perform the Upgrade 

After updating your configuration, upgrade your Refinery instances using your usual deployment method (such as Helm or Kubernetes). Deployment steps vary by environment, so follow your standard process to complete the upgrade.

Upgrading to Refinery 2.0 

Refinery 2.0 introduced a new YAML-based configuration format and included breaking changes that require migrating your configuration. Use the Configuration Conversion Tool to convert your Refinery 1.x files (JSON, TOML, and YAML format) to the Refinery 2.x YAML format and preserve your settings.

What’s New in Refinery 2.0 

Upgrading to Refinery 2.0 or later provides:

  • Improved .yaml-based configuration with better default values
  • New samplers: EMAThroughputSampler and WindowedThroughputSampler
  • Built-in configuration file validation
  • More accurate span counting (instead of trace counting) for dynamic samplers, making it easer to right-size sampling based on event volume
  • Improved metrics, including OpenTelemetry output
  • Support for emitting multiple metrics sources

Migrate Your Configuration 

To migrate from Refinery 1.x to 2.x:

  1. Gather this information before you start:

    • Whether you use Helm with Kubernetes to manage your Refinery instance
    • The file paths for your existing Refinery config and rules files
    • Any custom comments in your existing configuration files that you want to preserve

  2. Install the Configuration Conversion Tool.

  3. Convert your configuration files using one of these methods:

    • Convert Helm values file (if you use Kubernetes with Helm)
    • Convert config and rules files (if you don’t use Helm)

  4. Review the converted files and add any custom comments you want to preserve.

    Tip
    The Configuration Conversion Tool does not migrate custom comments from your existing files. Save any important comments separately and add them to your new files manually.

For detailed information about the tool’s capabilities, visit Configuration Conversion Tool.

Perform the Upgrade 

Once you’ve converted and reviewed your configuration files, upgrade Refinery using your existing deployment tool or process (for example, Helm, Kubernetes, or a custom deployment script). Follow your environment’s standard procedure to complete the upgrade.

Configuration Conversion Tool 

The Configuration Conversion Tool is a command line tool that helps you migrate Refinery configurations, validate configuration files, and generate local documentation. While primarily used for translating from Refinery 1.x to 2.x file format, you can also use it independently for validation and documentation tasks.

What It Does 

The Configuration Conversion Tool:

  • Converts Refinery 1.x configuration files (JSON, TOML, or YAML) to the Refinery 2.x YAML format.
  • Validates Refinery configuration files automatically during conversion.
  • Generates local documentation in Markdown format for offline reference.
Note
Refinery 2.x and 3.x use the same YAML configuration format. The conversion tool is only needed when upgrading from Refinery 1.x to 2.x or later.

How It Works 

The tool maps variables from their Refinery 1.x locations to the appropriate Refinery 2.x locations. Any non-default values carry over to the new format if still applicable. Example: If your Refinery 1.x config file specifies MetricsAPIKey in the HoneycombMetrics section, the tool creates a Refinery 2.x config.yaml file with that key listed as APIKey in the LegacyMetrics section.

The generated Refinery 2.x config.yaml files include helpful comments describing each configuration variable and its possible values.

Important
The tool doesn’t migrate custom comments from your existing configuration files. Save any important comments separately and add them to your new files manually.

Installing the Conversion Tool 

The configuration conversion tool is available as pre-built binaries for Linux, macOS, and Windows.

  • If you use Helm with Kubernetes: The configuration conversion tool installs automatically with Refinery 2.0 and later.
  • If you don’t use Helm: Download the configuration conversion tool from the Releases page in the Refinery GitHub repository. Find the single binary file for your operating system listed alongside each Refinery release, starting with Refinery 2.0.

Converting Configuration Files 

Follow the instructions for your installation method:

Convert your Refinery Helm chart to the Refinery 2.x format:

convert helm --input /path/to/values.yaml --output /path/to/values.yaml

Command structure:

  • convert: Command used to convert the specified input file
  • helm: Indicates the conversion template to apply
  • --input /path/to/values.yaml: Required path to your existing Refinery 1.x values.yaml source file. Adjust the file type and path as needed.
  • --output /path/to/values.yaml: Required path that designates where the tool will save your new Refinery 2.x values.yaml destination file

If you don’t use Helm, convert your Refinery config and rules files separately.

Convert Your Config File 

Convert your Refinery config file to the Refinery 2.x format:

convert config --input /path/to/config.toml --output /path/to/config.yaml

Command structure:

  • convert: Command used to convert the specified input file
  • config: Indicates the conversion template to apply
  • --input /path/to/config.toml: Required path to your existing Refinery 1.x config source file. Adjust the file type and path as needed.
  • --output /path/to/config.yaml: Required path that designates where the tool will save your new Refinery 2.x config destination file
Tip
The tool automatically detects the input file type based on the file extension. Override this by adding the --type option.

Convert Your Rules File 

Convert your Refinery rules file to the Refinery 2.x format:

convert rules --input /path/to/rules.toml --output /path/to/rules.yaml

Command structure:

  • convert: Command used to convert the specified input file
  • rules: Indicates the conversion template to apply
  • --input /path/to/config.toml: Required path to your existing Refinery 1.x rules source file. Adjust the file type and path as needed.
  • --output /path/to/config.yaml: Required path that designates where the tool will save your new Refinery 2.x rules destination file
Tip
The tool automatically detects the input file type based on the extension. Override this by adding the --type option.

Troubleshooting Conversion Errors 

The conversion tool automatically validates your Refinery file during conversion. If the tool finds issues, it stops and displays descriptive error messages.

For example, the following output could appear when multiple errors are encountered for the rules file:

Validation Errors in rules file:
  Within sampler dataset1: field DynamicSampler.UseTraceLength must be a bool
  Within sampler dataset1: field DynamicSampler.ClearFrequency (1) must be a valid duration like '3m30s' or '100ms'
  Within sampler dataset1: unknown field DynamicSampler.SamplerRate; did you mean SampleRate?
  Within sampler dataset1: field DynamicSampler.FieldList must be a string array but contains non-string 10
  Within sampler dataset1: the group DynamicSampler is missing its required field SampleRate

To continue with migration:

  1. Fix the errors in your Refinery 1.x file.
  2. Run the conversion tool again.

For help resolving specific errors, visit the Refinery documentation for configuration (config.yaml) and sampling (rules.yaml).

Command Line Options 

The tool supports these command line options:

-i,--input
Path to the Refinery 1.x file to convert (default: config.toml)
-o, --output
Path where the tool saves the converted Refinery 2.x file (default: stdout)
-t Y, --type=Y
Indicates that the tool should treat the input file as YAML format. Use when the file extension doesn’t indicate format.
-t T, --type=T
Indicates that the tool should treat the input file as TOML format. Use when the file extension doesn’t indicate format.
-t J, --type=J
Indicates that the tool should load the input file as JSON format. Use when the file extension doesn’t indicate format.
-h, --help
Display the help message.

Validating Configuration Files 

Manually validate your Refinery configuration files, such as in a Continuous Integration (CI) process. (Refinery automatically validates files during the convert command.)

To validate your Refinery config file against the Refinery 2.x format, use:

convert validate config --input config.yaml

To validate your Refinery rules file against the Refinery 2.x format, use:

convert validate rules --input rules.yaml

Command structure:

  • convert validate: Command used to validate the specified input file
  • config or rules: Indicates the appropriate validation template
  • --input: Required path to the file you want to validate

Generating Local Documentation 

Generate local Refinery documentation in Markdown format for offline reference.

To generate documentation for config.yaml, use:

convert doc config

To generate documentation for rules.yaml, use:

convert doc rules

To learn more, visit Refinery documentation for configuration (config.yaml) and sampling (rules.yaml).

Get Help 

Join #discuss-refinery in the Pollinators Community Slack to ask questions and learn more.

For Pro and Enterprise users, contact Support via support.honeycomb.io, or email at support@honeycomb.io

On this page: