Skip to main content
Ent+
This feature is available as an add-on for the Honeycomb Enterprise plan. Please contact your Honeycomb account team for details.

​
Description

The Deduplicate Logs processor can be used to deduplicate logs over a time range and emit a single log with the count of duplicate logs. Logs are considered duplicates if the following match:
  • Severity
  • Log Body
  • Resource Attributes
  • Log Attributes

​
Supported Types

MetricsLogsTraces
✓

​
Configuration Table

ParameterTypeDefaultDescription
interval*int10The interval in seconds on which to aggregate logs. An aggregated log will be emitted after the interval passes.
log_count_attribute*stringlog_countThe name of the count attribute of deduplicated logs that will be added to the emitted log.
timezone*stringUTCThe timezone of the first_observed_timestamp and last_observed_timestamp log attributes that are on the emitted log.
include_fieldsstringsA list of fields to include in duplicate matching. Fields can be from the log body or attributes. This option is mutually exclusive with exclude_fields. More details can be found here.
exclude_fieldsstringsA list of fields to exclude from duplicate matching. Fields can be excluded from the log body or attributes. These fields will not be present in the emitted log. This option is mutually exclusive with include_fields. More details can be found here.
* required field

​
include_fields Parameter

The include_fields parameter allows the user to remove fields from being considered when looking for duplicate logs. Fields can be included from either the body or attributes of a log. Though the entire body cannot be included. Nested fields can be specified by delimiting each part of the path with a .. If a field contains a . as part of its name it can be escaped by using \.. Below are a few examples and how to specify them:
  • Include timestamp field from the body -> body.timestamp
  • Include a log.file.name field from the log attributes -> attributes.log\.file\.name
  • Include a nested ip field inside a src attribute -> attributes.src.ip

​
exclude_fields Parameter

The exclude_fields parameter allows the user to remove fields from being considered when looking for duplicate logs. Fields can be excluded from either the body or attributes of a log. Though the entire body cannot be excluded. Nested fields can be specified by delimiting each part of the path with a .. If a field contains a . as part of its name it can be escaped by using \.. Below are a few examples and how to specify them:
  • Exclude timestamp field from the body -> body.timestamp
  • Exclude a host.name field from the log attributes -> attributes.host\.name
  • Exclude a nested ip field inside a src attribute -> attributes.src.ip

​
Example Configuration

​
Basic Configuration

Setting a custom log_count_attribute and timezone while deduplicating logs on a 60 second interval.

​
Web Interface

Honeycomb Docs - Deduplicate Logs - image 1

​
Standalone Processor

apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  id: log-dedup
  name: log-dedup
spec:
  type: log_dedup
  parameters:
    - name: interval
      value: 60
    - name: log_count_attribute
      value: 'dedup_count'
    - name: timezone
      value: 'America/Los_Angeles'

​
Exclude Fields

This example shows the addition of exclude_fields. More information on exclude_fields can be found here.

​
Web Interface

Honeycomb Docs - Deduplicate Logs - image 2

​
Standalone Processor

apiVersion: bindplane.observiq.com/v1
kind: Processor
metadata:
  id: exclude-fields
  name: exclude-fields
spec:
  type: log_dedup
  parameters:
    - name: interval
      value: 10
    - name: log_count_attribute
      value: 'log_count'
    - name: timezone
      value: 'UTC'
    - name: exclude_fields
      value:
        - 'attributes.timestamp'
        - 'body.time'
        - 'attributes.log\.file\.name'