We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Using Triggers

Triggers let you receive notifications when your data in Honeycomb crosses the thresholds you configure. The graph on which to alert is as flexible as a Honeycomb query, which helps reduce false positives due to known errors.

When a trigger fires, you’ll be notified via the configured method. Currently supported are PagerDuty, Slack, Webhooks, and Email. The notification includes a link back to the graph showing you the current status, providing a jumping off point for further investigation.

Triggers may be scheduled to run on one minute intervals, from once per minute to once per day (1440 minutes.) The frequency period also determines the time range for each execution of the trigger.

For example: a trigger every 15 minutes will examine the past 15 minutes of activity to compute the values against which the thresholds are measured.

Important: trigger frequency must be specified in whole minutes, from 1 to 1440. Decimal values are truncated to the preceding full minute. (3.6 becomes 3.)

For this example, we want to know whenever the 95th percentile of our API server’s requests exceeds 30ms, but we want to exclude the /poll endpoint because it has long-held connections, which pollute the data by being artificially high.

Create a Trigger

Start the trigger creation process by building and running a query. Select the three-dot overflow menu and click “Make Trigger”.

The Query Builder

Configuration

Fill in the details for the trigger. Both the Name and Description will be included in notifications about the trigger. Make sure the name describes clearly what has happened, while the description should indicate next steps or include links back to documentation, so that the person who receives the alert will know how to respond.

Define New Trigger

The sample graph displays the most recent 16 periods for your query (with a granularity equal to the period length) to help choose an appropriate threshold. For example, the graph for a 5 minute frequency shows the previous 80 minutes with a 5 minute granularity.

Define Threshold

The Threshold indicates what condition generates a notification, and the Frequency determines how often to check for that condition. Consider what is normal within your frequency window so notifications only capture conditions worth alerting on.

The trigger will notify all recipients listed when the measured value crosses the configured threshold.

Add Recipients

By default, you will be able to enter email recipients. Additional recipients will be available in the dropdown after you have configured them on a team-level basis.

Trigger Recipient Integrations (Slack, Pagerduty, Webhooks)

You can configure PagerDuty, Slack, or Webhooks as trigger recipients that can be used by anyone on your Honeycomb team. If you are in the process of creating a trigger, you can click “Configure Integrations” from within the trigger editing page to set them up in your team settings page.

Configure Recipients

You can also find this page by clicking on your user icon in the navigation sidebar > Team settings > Integrations. Here you can add, edit, or remove your team-level trigger recipients. Deleting an Integration from here will remove them from all associated triggers.

Configure team-level trigger recipients

Slack

Allow Honeycomb to connect to your Slack team to send triggers to your Slack channels, with nifty features such as link unfurling that will show you a preview of your Honeycomb query result graphs. Only one of your Slack team’s members needs to authorize Honeycomb in order to send triggers to public channels. Team members who want to send triggers to their private channels or themselves must authorize Slack on an individual basis.

Note: previously, our Slack trigger recipients used Slack’s Incoming Webhooks. These recipients will still work, but you will no longer be able to add new recipients in this way.

PagerDuty

PagerDuty’s API Integration docs describe how to create a generic API integration to PagerDuty. Following those steps will give you an API key that you’ll enter in the Trigger Recipient configuration form.

Webhooks

You can specifiy Webhooks for Honeycomb to send JSON payloads to upon trigger firing, to build your custom integrations. A webhook is an HTTP endpoint running within your infrastructure to which Honeycomb will send notifications of the trigger changing state. The content will include an authentication header and the result of the trigger in JSON in the body of the webhook.

The API for webhook notifications are described via an example webhook implementation that can consume webhook notifications.

Activate Trigger

When you save the trigger, it is immediately active and will trigger at the next frequency interval (such as on the next 5 minute interval for a 5 minute frequency.)

List and Delete

You can find the triggers belonging to a dataset by going into the settings page for the dataset, then clicking the “Triggers” tab.

Datasets Page

You’ll see a full list of the triggers active for the dataset. You’ll be able to click through and edit each trigger from this list.

List of Triggers within a Dataset page

To delete a trigger, scroll to the bottom of the edit page.

Considerations

Queries done for triggers run your selected calculation over your configured frequency period. COUNTs, for example, will be the total count for that period, not a per-second count. Averages and percentiles are likewise covering the entire period—so to detect spikes, it is better to use MAX instead of AVG over a period. Another alternative is to use a COUNT with a filter restricting the set to the threshold you’re interested in—for example, you could count the number of events over 100ms and use that with a threshold instead of asking for the average to exceed 100ms.

Best practices