Transform Your Data

In Honeycomb, you can define unique custom fields that help you flexibly transform your data without making permanent changes to your dataset.

What are Custom Fields?

Otherwise known as Derived Columns, custom fields are computed properties that are calculated by a formula.

The field’s value is based on the result of the formula, which can contain functions, mathematical and/or logical operations on other field values, and constants and literals–similar to expressions defined in a spreadsheet. To learn more about syntax and available functions, visit Derived Column Formula Reference.

Transforming Data with Custom Fields

In Honeycomb, you can use custom fields to create new data fields based on existing ones, so you can perform advanced data transformation. Then, you can use custom fields in Honeycomb just as you would use other fields in your data, such as when building queries, defining Triggers, and creating Service Level Objectives (SLOs).

When transforming data, you can use custom fields to:

cleanse data
create boolean flags and indicators
categorize data
calculate a new value, so you can express an event detail in a more human-readable way
transform your source data values to Honeycomb standard values (for example, log severities)

Tip

Honeycomb calculates custom fields dynamically whenever you run a query, which ensures the value of your field is always based on the latest data.

Tip

When planning your transformations, remember that Honeycomb does not allow you to nest custom fields.

Cleansing Data

If you have non-standard data, you can use custom fields to cleanse or format fields values. This example standardizes text in a field by converting it to all lowercase:

TO_LOWER($service.name)

Creating Boolean Flags or Indicators

You can use custom fields to create flags for easier filtering and analysis. This example creates a flag for HTTP statuses greater than or equal to 400:

IF(GTE($http_status, 400), 1, 0)

Categorizing Data

You can use custom fields to classify data based on conditions for easier segmentation. This example classifies API version based on URL fragment:

IF(CONTAINS($url, "/v1/"), "api_v1", "api_v2")

Calculating New Values

You can use custom fields to calculate values based on existing fields to create new metrics. This example calculates event latency and converts the unit from seconds to minutes.

DIV(SUB(INGEST_TIMESTAMP(),EVENT_TIMESTAMP()),60)

Transforming Source Data Values into Honeycomb Standard Values

In Honeycomb, certain visualizations rely on Honeycomb-specific, standard values. For example, the full experience of log visualization relies on Honeycomb standard severity values being populated.

You can use custom fields to transform your source data values to Honeycomb standard values without making permanent changes to your dataset.

Transform Source Severities into Honeycomb Standard Severities

If you are sending log source data to Honeycomb and the data you are sending contains severities other than the standard log severities expected by Honeycomb, Honeycomb visualizations that depend on severity may not populate. To solve this issue, use a custom field to transform any non-standard severities to Honeycomb standard log severities.

Note

The standard Honeycomb log severities include info, error, warn, fatal, trace, debug, and unspecified.

Example

Scenario: You have a source field named my.severity that contains values of a, b, c, d, e, f, and g. You want the values to map like so:

Your Severity	Honeycomb Standard Log Severity
`a`	`info`
`b`	`error`
`c`	`warn`
`d`	`fatal`
`e`	`trace`
`f`	`debug`
`g`	`unspecified`

Any other severity values should be recognized as non-standard and labeled as Other by Honeycomb.

Solution: Use a Derived Column to parse your severity values and transform them to Honeycomb standard log severity values.

Create a Derived Column, and enter the following function in the Derived Column Editor:

IF(
  REG_MATCH($my.severity, "a"), "info",
  REG_MATCH($my.severity, "b"), "error",
  REG_MATCH($my.severity, "c"), "warn",
  REG_MATCH($my.severity, "d"), "fatal",
  REG_MATCH($my.severity, "e"), "trace",
  REG_MATCH($my.severity, "f"), "debug",
  REG_MATCH($my.severity, "g"), "unspecified",
  "Other"
)

Use your Derived Column as the source field, and map it to the Logs: Severity dataset field when you map your data.

Your severities will now map to Honeycomb standard log severities, and visualizations that depend on severity should populate.

Creating Custom Fields

Important

You can create custom fields at both the Environment and the Dataset level. All custom fields are applied at the Dataset level, but Environment-level custom fields are shared and available for you to use across all Datasets in your Environment.

To create a custom field:

In the Honeycomb UI, select the Environments label from the navigation sidebar, then select Manage Environments.
In the list, locate the Environment to which you would like to add a Derived Column, and select its name to view the available settings.
Navigate to the Schema view.
Select Add New Derived Column.
In the modal, enter a Display Name, which will show as the field name in the Query Builder.

Tip

Enter a display name that is unique across the containing Environment. Your Derived Column name should not match the name of any other Derived Column or any other field in any Dataset contained within the Environment.

Although Honeycomb tries to prevent duplicate field names, they can still occur. To learn more about behaviors related to name collision and solutions, visit Common Issues with Queries: Derived Columns.
In the editor, define the formula for your Derived Column. To learn more about syntax and available functions, and to explore some example formulas, visit Derived Column Formula Reference.

Hover over any syntax errors (red underlines or red triangles) for assistance with correcting them.
Select Save.