1. Docs
  2. Troubleshoot
  3. Recommended Migrations
  4. Refinery 2.0

Migrate to Refinery 2.0 and later

Starting with Refinery 2.0, there is a breaking change that requires migration to take advantage of new and improved Refinery features. Use our configuration conversion tool to convert your existing configuration and to preserve important settings.

Benefits of upgrading your Refinery instance to 2.0 or later include:

  • Improved .yaml-based configuration with better default values
  • New samplers: EMAThroughputSampler and WindowedThroughputSampler
  • Configuration file validation
  • Having all dynamic samplers now correctly count spans, not traces, to determine the sampling rate, which makes it easier to right-size sampling based on event volume
  • Improved metrics, including output as OpenTelemetry
  • Support for emitting multiple metrics sources

Configuration Conversion Tool 

Our configuration conversion tool is a command line tool.

The tool maps the variables in your existing Refinery 1.x config and rules files and translates them to the .yaml format. Any non-default values from their Refinery 1.x location are copied to the appropriate Refinery 2.x location if still applicable. For example, if the HoneycombMetrics section in the Refinery 1.x config file specified MetricsAPIKey, then the Refinery 2.x config.yaml file will list that key as the APIKey name in the LegacyMetrics section.

Supported Refinery 1.x file formats for this tool are .json, .toml, and .yaml.

The new Refinery 2.x config.yaml files also includes comments that describe each configuration variable and its possible values.

Tip
Any custom comments held within your existing configuration files will not be migrated to the Refinery 2.x .yaml files.

In addition, the configuration conversion tool has advanced features and command line options.

Migration Process 

Migration Preparation 

Before installing the configuration conversion tool:

  1. Determine if Helm with Kubernetes is used to manage your existing Refinery instance. This determines the conversion tool path to use.
  2. Identify the file paths for your existing Refinery config and rules files
  3. Identify any custom comments to preserve within the existing configuration files. These custom comments will not migrate and preservation requires manual effort to include in the new file(s).

Having this information identified ahead of time helps to ensure a smoother migration process.

Install 

The configuration conversion tool is available as built binaries for Linux, macOS, and Windows.

If using Helm with Kubernetes to install and upgrade Refinery, the configuration conversion tool is automatically available with Refinery 2.0 and later.

If not using Helm, download the configuration conversion tool from the Releases page of the Refinery GitHub repository. It is available as a single binary file for each operating system alongside each Refinery release, starting with Refinery 2.0.

Run Configuration Conversion Tool 

After installation (if necessary), run the Configuration Conversion Tool for your installation option:

For Helm users, convert your Refinery Helm chart to the Refinery 2.0 and later format by using:

convert helm --input /path/to/values.yaml --output /path/to/values.yaml

where:

  • convert is the command to convert the designated --input file
  • helm is the conversion template to apply
  • --input is a required command line option to designate the file path of your source file
  • /path/to/values.yaml is the file path of your existing Refinery 1.x values.yaml file
  • --output is a required command line option to designate the file path of your new Refinery 2.x values.yaml destination file
  • /path/to/values.yaml is the file path of the designated --output destination file

If not a Helm user, use the following two commands to convert your Refinery config and rules files.

Convert Config 

To convert your Refinery config file to the Refinery 2.x format, use:

convert config --input /path/to/config.toml --output /path/to/config.yaml

where:

  • convert is the command to convert the designated --input file
  • config is the conversion template to apply
  • --input is a required command line option to designate the file path of your source file
  • /path/to/config.toml is the file path of your existing Refinery 1.x --input source file. Your migration situation may involve a different file type or path
  • --output is a required command line option to designate the file path of the new Refinery 2.x .yaml destination file
  • /path/to/config.yaml is the file path of the designated --output destination file
Tip
Configuration Conversion Tool automatically attempts to determine the file type of the input file based on the extension, but you can override by including the --type option.

Convert Rules 

To convert your Refinery rules file to the Refinery 2.x format, use:

convert rules --input /path/to/rules.toml --output /path/to/rules.yaml

where:

  • convert is the command to convert the designated --input file
  • rules is the conversion template to apply
  • --input is a required command line option to designate the file path of your source file
  • /path/to/rules.toml is the file path of your existing Refinery 1.x --input source file. Your migration situation may involve a different file type or path
  • --output is a required command line option to designate the file path of your .yaml destination file
  • /path/to/rules.yaml is the file path of the designated --output destination file
Tip
Configuration Conversion Tool automatically attempts to determine the file type of the input file based on the extension, but you can override by including the --type option.

Troubleshooting 

As a part of the configuration conversion process, the Refinery file is automatically validated. If one or more issues are encountered, the conversion process stops and descriptive error message(s) appear in the command line output.

For example, the following output appears with multiple error messages for the rules file:

Validation Errors in rules file:
  Within sampler dataset1: field DynamicSampler.UseTraceLength must be a bool
  Within sampler dataset1: field DynamicSampler.ClearFrequency (1) must be a valid duration like '3m30s' or '100ms'
  Within sampler dataset1: unknown field DynamicSampler.SamplerRate; did you mean SampleRate?
  Within sampler dataset1: field DynamicSampler.FieldList must be a string array but contains non-string 10
  Within sampler dataset1: the group DynamicSampler is missing its required field SampleRate

To continue with migration, correct the errors found in your Refinery 1.x file and re-run the configuration conversion tool.

In troubleshooting, reference our online Refinery documentation for configuration (config.yaml) and sampling (rules.yaml).

Advanced Conversion Tooling 

The configuration conversion tool features additional command line options and tool features as documented below.

Command Line Options 

The Refinery configuration conversion tool has the following command line options:

-i,--input
The Refinery 1.x file to read (default: config.toml)
-o, --output
The updated Refinery 2.x file to write (default: stdout)
-t Y, --type=Y
Loads the input file as YAML format (in case file extension does not work)
-t T, --type=T
Loads the input file as TOML format (in case file extension does not work)
-t J, --type=J
Loads the input file as JSON format (in case file extension does not work)
-h, --help
Use to show the help message

Additional Tool Features 

In addition to file conversion, the configuration conversion tool also features the ability to manually validate and generate local documentation.

Validate 

Use the following commands to manually validate the Refinery configuration files, such as manual validation in a Continuous Integration (CI) process.

Refinery automatically validates as a part of the convert command.

To validate your Refinery config file against the Refinery 2.x format, use:

convert validate config

To validate your Refinery rules file against the Refinery 2.x format, use:

convert validate rules

For example, this command validates the config.yaml config file:

convert validate config --input config.yaml

where:

  • convert validate is the command to validate the designated --input file
  • config is the validation template to apply
  • --input is a required command line option to designate the file path of your source file
  • config.yaml is the file path of the designated --input source file

Generate Documentation in Markdown Form 

To generate local Refinery documentation in markdown (.md) form for config.yaml, use:

convert doc config

To generate local Refinery documentation in markdown (.md) form for rules.yaml, use:

convert doc rules

Also, refer to our online Refinery documentation for configuration (config.yaml) and sampling (rules.yaml).

Questions and Support 

Join the #discuss-refinery in our Pollinators Community Slack to ask questions and learn more.

For Pro and Enterprise users, contact Support via support.honeycomb.io, or email at support@honeycomb.io

On this page: