You can create a trigger within the Triggers page or while using the Query Builder.
Select the Triggers icon in the left navigation bar to reach the Triggers page. From the Triggers page:
Creating a new trigger from the Triggers page will require entering a query during trigger configuration.
From the Query Builder screen:
For this example, we want to know whenever our cart has an error that is considered “slow” and to have the results grouped by userid, http.url, and requestID.
To configure the trigger, define the trigger details, trigger alert threshold, and notification preferences in the next screen.
Both the Name and Description will be included in notifications about the trigger. Ensure the name describes clearly what has happened, while the description should indicate next steps or include links back to documentation, so that the person who receives the alert will know how to respond.
After defining the trigger details within the first section, the next section displays a sample graph that shows how the trigger query and the trigger threshold components interact. When creating a new Trigger from Query Builder, the sample graph will appear automatically. When creating a new Trigger from the Triggers page, you must enter a query before the sample graph appears and the Threshold field in the Alerts section populates.
The sample graph displays the trends for your query with the most recent 16 periods as indicated by markers. Set the period length with Duration in the Alerts section to help choose an appropriate threshold. The default Duration value is 15 minutes. Use Frequency in the Alerts section to control the frequency of the query run. The frequency of the query is set at 15 minutes by default. For example, the sample graph for a 30 minute frequency with a 120 minute duration shows the previous 1920 minutes (or 32 hours).
Next, define the threshold for the trigger alert.
The Threshold indicates what condition generates a notification. By default, a notification generates whenever the condition meets the threshold or resolves once below the threshold.
About Triggered Groups: If you have specified fields in the GROUP BY clause of a trigger, then the trigger will notify all recipients when any new group crosses the trigger threshold.
For example, if a trigger is already in a triggered state, and any new group surpasses the trigger threshold, the trigger will again notify all recipients and include the new groups that have triggered the alert.
Also, set the number of times the Threshold, or trigger condition, should be met consecutively before alerting you.
Use Send an alert after the threshold has been met x times to enter this value.
This value defaults to 1 and cannot be greater than 5.
How often the trigger condition is evaluated based on its Frequency value.
For example, if the number of times a trigger’s threshold has been met is 3 before alerting and the trigger’s frequency is 5 minutes, then this trigger alerts when its threshold has been met for the past 15 minutes, or 3 cycles of 5 minutes.
Choose between “Limited alerts” and “Continuous alerts”.
Select “Limited alerts” for the default behavior of alerting once when crossing the threshold and once upon resolution.
Select “Continuous alerts” for an alert to be sent every time the condition is met or exceeds the threshold. When this setting is enabled, no resolved alert is sent. Use “Continuous alerts” when:
For example, if a trigger has specified fields in its GROUP BY clause, and “Continuous Alerts” is selected, then the trigger will notify all recipients when any new group crosses the trigger threshold, or if any group still exceeds the threshold. If one or more groups resolve, no resolved alert will be sent.
The Duration determines what time range of data that the trigger will check. The default Duration value is 15 minutes. You will see an Event Latency History graph next to the duration field. This graph describes the maximum and average amount of delay between the timestamp on the event and when it reached Honeycomb. You can use this data to help choose a duration that captures all your events, even if they are delayed.
For example, if the average event latency is 2 minutes, and you want to run your trigger every 5 min, choose a 7 minute duration to ensure that delayed events are captured by the trigger. Please note that if your traces span a long time frame, you may see high latency in this chart, even though the traces are arriving as soon as they complete.
The Frequency determines how often, in minutes, to check for the Threshold condition. The default Frequency value is 15 minutes. Consider what is normal within your frequency window so notifications only capture conditions worth alerting.
1 to 1440.
Decimal values are truncated to the preceding full minute.
(3.6 becomes 3.) Honeycomb allows you to specify a scheduled window in which the trigger will run. For example, you may have a situation that you need to be alerted on Monday through Friday, but not on the weekends.
To enable, slide the Custom scheduling toggle to the right and specify the time range and days that the trigger should run. Note that the start time and end time must be provided in Coordinated Universal Time (UTC).
The trigger will notify all listed recipients when the measured value crosses the configured threshold. No limitation exists for the number of recipients. By default, Honeycomb will send an alert to recipients once, when the trigger crosses the configured threshold or the Triggered state, and then send a resolved alert once the trigger is back in an OK state.
To add a new recipient, select Add Recipient. Use Go to Integration Center to configure additional trigger recipient integration options, like Slack, PagerDuty, Microsoft Teams, and Webhooks.
After selecting Add Recipient, a form will appear with Recipient options in a dropdown list. By default, you can select Notify by Email and enter email recipients. Additional integration options, like Slack, PagerDuty, Microsoft Teams, and Webhooks, can be selected once configured.
Finally, select Create Trigger to save your trigger configuration.
Once saved, the trigger is immediately active and will run at the next frequency interval, such as on the next 5 minute interval for a 5 minute frequency. You can enable or disable a trigger by editing the trigger and selecting the Enable or Disable option.
You can create a Trigger from a template. Trigger Templates are visible in the right sidebar when creating a new Trigger. Selecting a template populates the Name, Description, and Query in the Trigger form.
Trigger Templates reference fields that are set in your Dataset Definitions. If you select a template that references a field mapping that does not exist in your data, a warning message displays above the Trigger Query section that lists the missing Dataset Definition(s), as seen in the example below. To resolve the issue and fully use the Trigger Template, add the missing field and/or update your Dataset Definitions to include the missing Dataset Definition(s).
| Template | Trigger Name | Trigger Description | Dataset Definitions Required |
|---|---|---|---|
| Latency | Latency is too high | This trigger notifies us if the average duration of root spans is higher than 500ms over the last 15 minutes. In the future, consider using P90 or P95 operators to handle outliers more reliably. | Span Duration Parent Span ID |
| HTTP Errors | Too many HTTP request errors | This trigger notifies us if there are any 400 or 500 level HTTP status requests | HTTP Status Code Parent Span ID |
| General Errors | Too many errors | This trigger notifies us if there are errors | Error |