Transform Your Data

In Honeycomb, you can define unique calculated fields that help you flexibly transform your data without making permanent changes to your dataset.

What are Calculated Fields? 

Otherwise known as Derived Columns, calculated fields are computed properties that are calculated by a formula.

The field’s value is based on the result of the formula, which can contain functions, mathematical and/or logical operations on other field values, and constants and literals–similar to expressions defined in a spreadsheet. To learn more about syntax and available functions, visit Calculated Field Formula Reference.

Transforming Data with Calculated Fields 

In Honeycomb, you can use calculated fields to create new data fields based on existing ones, so you can perform advanced data transformation. Then, you can use calculated fields in Honeycomb just as you would use other fields in your data, such as when building queries, defining Triggers, and creating Service Level Objectives (SLOs).

When transforming data, you can use calculated fields to:

  • cleanse data
  • create boolean flags and indicators
  • categorize data
  • calculate a new value, so you can express an event detail in a more human-readable way
  • transform your source data values to Honeycomb standard values (for example, log severities)
Tip
Honeycomb calculates calculated fields dynamically whenever you run a query, which ensures the value of your field is always based on the latest data.
Tip
When planning your transformations, remember that Honeycomb does not allow you to nest calculated fields.

Cleansing Data 

If you have non-standard data, you can use calculated fields to cleanse or format fields values. This example standardizes text in a field by converting it to all lowercase:

TO_LOWER($service.name)

Creating Boolean Flags or Indicators 

You can use calculated fields to create flags for easier filtering and analysis. This example creates a flag for HTTP statuses greater than or equal to 400:

IF(GTE($http_status, 400), 1, 0)

Categorizing Data 

You can use calculated fields to classify data based on conditions for easier segmentation. This example classifies API version based on URL fragment:

IF(CONTAINS($url, "/v1/"), "api_v1", "api_v2")

Calculating New Values 

You can use calculated fields to calculate values based on existing fields to create new metrics. This example calculates event latency and converts the unit from seconds to minutes.

DIV(SUB(INGEST_TIMESTAMP(),EVENT_TIMESTAMP()),60)

Transforming Source Data Values into Honeycomb Standard Values 

In Honeycomb, certain visualizations rely on Honeycomb-specific, standard values. For example, the full experience of log visualization relies on Honeycomb standard severity values being populated.

You can use calculated fields to transform your source data values to Honeycomb standard values without making permanent changes to your dataset.

Transform Source Severities into Honeycomb Standard Severities 

If you are sending log source data to Honeycomb and the data you are sending contains severities other than the standard log severities expected by Honeycomb, Honeycomb visualizations that depend on severity may not populate. To solve this issue, use a calculated field to transform any non-standard severities to Honeycomb standard log severities.

Note
The standard Honeycomb log severities include info, error, warn, fatal, trace, debug, and unspecified.
Example

Scenario: You have a source field named my.severity that contains values of a, b, c, d, e, f, and g. You want the values to map like so:

Your Severity Honeycomb Standard Log Severity
a info
b error
c warn
d fatal
e trace
f debug
g unspecified

Any other severity values should be recognized as non-standard and labeled as Other by Honeycomb.

Solution: Use a Calculated Field to parse your severity values and transform them to Honeycomb standard log severity values.

  1. Create a Calculated Field, and enter the following function in the Editor:

    IF(
      REG_MATCH($my.severity, "a"), "info",
      REG_MATCH($my.severity, "b"), "error",
      REG_MATCH($my.severity, "c"), "warn",
      REG_MATCH($my.severity, "d"), "fatal",
      REG_MATCH($my.severity, "e"), "trace",
      REG_MATCH($my.severity, "f"), "debug",
      REG_MATCH($my.severity, "g"), "unspecified",
      "Other"
    )
    
  2. Use your Calculated Field as the source field, and map it to the Logs: Severity dataset field when you map your data.

Your severities will now map to Honeycomb standard log severities, and visualizations that depend on severity should populate.

Creating Calculated Fields 

Important
You can create calculated fields at both the Environment and the Dataset level. All calculated fields are applied at the Dataset level, but Environment-level calculated fields are shared and available for you to use across all Datasets in your Environment.

To create a calculated field:

  1. In the Honeycomb UI, select the Environments label from the navigation sidebar, then select Manage Environments.

  2. In the list, locate the Environment to which you would like to add a Calculated Field, and select its name to view the available settings.

  3. Navigate to the Schema view.

  4. Select Add New Calculated Field.

  5. In the modal, enter a Display Name, which will show as the field name in the Query Builder.

    Tip

    Enter a display name that is unique across the containing Environment. Your Calculated Field name should not match the name of any other Calculated Field or any other field in any Dataset contained within the Environment.

    Although Honeycomb tries to prevent duplicate field names, they can still occur. To learn more about behaviors related to name collision and solutions, visit Common Issues with Queries: Calculated Fields.

  6. In the editor, define the formula for your Calculated Field. To learn more about syntax and available functions, and to explore some example formulas, visit Calculated Field Formula Reference.

    Hover over any syntax errors (red underlines or red triangles) for assistance with correcting them.

  7. Select Save.