Gate Deployments with GitHub Actions Deployment Protection Rules


GitHub Actions Deployment Protection Rules are automatic controls that can use data from external tools to gate deployments in Actions workflows. Honeycomb provides a deployment protection rule that lets you use Honeycomb query data in tandem with this feature.

The Honeycomb Deployment Protection Rule lets you programmatically define service expectations in your GitHub code to determine when a deployment should proceed. For each GitHub environment in your deployment, you define a Honeycomb query to run in a specific dataset and an allowable threshold for the results.

This rule provides a degree of flexibility that is designed to accommodate numerous use cases. For example, use the Honeycomb Deployment Protection Rule to:

  • gate deployments based on error rates introduced by the current build.
  • run automated performance checks in blue/green style deployments.
  • drive canary release patterns.
  • check SLI error rates in staging before promoting to production.
  • prevent automated deployments during an ongoing incident.
  • ensure that deployments are sensitive to the current service status in any environment.
Note
The Honeycomb Deployment Protection Rule is available to all Honeycomb customers. However, you must be a GitHub Enterprise Cloud customer to use any GitHub Actions Deployment Rules.

How It Works 

When enabled, the Honeycomb Deployment Protection Rule runs as a check prior to any deployment actions. It determines whether a deployment into the target GitHub environment is allowed to proceed.

Honeycomb deployment protection rules consist of a Honeycomb query, a threshold, and an operator (>, <, or =). When a deployment is requested in your Actions workflow, the rule sends its payload to Honeycomb for evaluation, where it asks if the deployment is allowed to proceed. After Honeycomb completes the query evaluation, it returns a pass/fail response.

All deployment protection rules are configured in a .honeycomb.yaml file that must be checked and located in the root directory of your GitHub repository. Otherwise, Honeycomb does not have permission to access this file. The .honeycomb.yaml file can contain a configuration block for each GitHub environment that the repository is deployed.

how-it-works.png

Prerequisites and Limitations 

Installation and Setup 

Install the Honeycomb GitHub App 

The Honeycomb Deployment Protection Rule is provided via the Honeycomb GitHub App.

To install:

  1. Navigate to the Honeycomb GitHub App.
    1. If using the Honeycomb US instance accessed via ui.honeycomb.io
    2. If using the Honeycomb EU instance accessed via ui.eu1.honeycomb.io
  2. Select Install.
  3. Then, select which organization to install the Honeycomb GitHub App.
    Note
    The GitHub App installation flow requires a GitHub Owner to grant authorization that allows Honeycomb access to your GitHub organization and its repositories.
  1. Authorize the Honeycomb GitHub App to your GitHub organization. An example of requested permissions appears below.

      GitHub app authorization modal  

  2. Once the GitHub authorization is complete, you are taken to Honeycomb to continue the installation process.

  3. In Honeycomb, select the Honeycomb team to connect.

    Note
    The Honeycomb team selection flow requires Honeycomb Team Owner permissions to complete.

  4. Then, confirm which team’s Honeycomb environments and datasets can be queried from your deployment protection rules.

Refer to Troubleshooting if you encounter any issues during installation.

Enable the Honeycomb Deployment Protection Rule in Each GitHub Environment 

After installing the Honeycomb GitHub App, it must be enabled as an environment protection rule for your various GitHub environments.

Note
GitHub Actions Deployment Protection Rules apply at the GitHub environment level, not at the repository level. Therefore, you must repeat this step for each GitHub environment where you wish to use the Honeycomb Deployment Protection Rule.

To enable:

  1. Navigate to your GitHub repository.
  2. Under Settings > Environments, select the GitHub environment by its name.
  3. Next, under Deployment protection rules, select the Honeycomb.io option. Optionally, select Allow administrators to bypass configured protection rules to allow GitHub Owners the ability to force a deployment when a deployment protection rule is in a failure state.
  4. Select Save Protection Rules to continue.

  Enable Honeycomb Deployment Protection Rule in GitHub  

Refer to GitHub documentation for more about environment protection rules.

Configure Honeycomb Deployment Protection Rules 

You must create a .honeycomb.yaml file in the root of your repository to configure the queries and thresholds that act as a deployment protection rule for each GitHub environment. The YAML file must have a configuration block for every environment with a deployment protection rule enabled; otherwise, the stage fails.

Note
The .honeycomb.yaml file must be in the root directory of your repository. Honeycomb cannot access this file in a subdirectory.

If the query defined as a deployment protection rule evaluates within the acceptable threshold, it “passes” and the deployment continues. Otherwise, it “fails” and the deployment is blocked.

Honeycomb supports a query specification for defining queries via JSON. Queries for Honeycomb deployment protection rules can be composed directly as JSON in your .honeycomb.yaml file. Guidance and resources exist for writing deployment protection queries.

Below is an example of a standard .honeycomb.yaml file.

version: 1

honeycomb_team: my-team # the honeycomb team where queries for all deployment protection rules runs

deployment_protection_rules:
  staging: # name of github environment where a deployment occurs
    queries:
      - honeycomb_environment: qa # honeycomb environment where the query runs
        honeycomb_dataset: my-cool-application # (optional, but necessary when using Honeycomb Classic) When left empty, an environment-wide Honeycomb query runs, which is not supported by Honeycomb Classic. 
        spec: &queryspec '{
            "time_range": 1800,
            "calculations": [
              {
                  "op": "COUNT"
              }
            ],
            "filters": [
              {
                  "column": "status_code",
                  "op": "=",
                  "value": "500"
              },
              {
                  "column": "build_id",
                  "op": "=",
                  "value": ${GITHUB_RUN_ID}  # NOTE: when deployment protection rules run, they can map the special GITHUB_RUN_ID variable
              }
            ],
            "filter_combination": "AND"
          }'
        threshold:
          operator: '>'
          value: 3

  production: # name of github environment where a deployment occurs
    queries:
      - honeycomb_environment: staging # honeycomb environment where the query runs
        honeycomb_dataset: my-cool-application # (optional, but necessary when using Honeycomb Classic) When left empty, an environment-wide Honeycomb query runs, which is not supported by Honeycomb Classic. 
        spec: *queryspec # example of a YAML alias to reuse a query spec created for a different deployment protection rule
        threshold:
          operator: '>'
          value: 1

where:

  • honeycomb_team must be the team where queries for all deployment protection rules runs. This is checked against the team authorized when installing the GitHub App.
  • GITHUB_RUN_ID is a special variable interpolated by the GitHub App as the unique identifier of the GitHub Actions workflow run.
Note

To configure GitHub Deployment Protection Rules against Honeycomb Classic:

  • Specify $classic$ as the environment slug
  • Specify the dataset (because environment-wide queries are not compatible with Honeycomb Classic)

Writing Deployment Protection Queries 

When writing Deployment Protection Queries, remember the following:

  • They operate similar to Triggers. - The same queries and thresholds used to define Triggers can be used in deployment protection rules. Refer to our Trigger Examples for inspiration.

  • Use the Query Builder UI to generate Query Spec JSON. - As an alternative to composing JSON directly, you can use Honeycomb’s Query Builder to create the JSON for you.

    1. Compose the query you wish to use as a deployment protection rule in the Query Builder and select Run Query.

    2. After you are satisfied with the query results that appear, select the three-dot overflow menu, located to the left of Run Query. Then, select View Query Definition for API.

        options-query-builder.png  

    1. A modal appears with the Query JSON. Use the copy button to copy the content, and paste it into the spec of your .honeycomb.yaml file.

        copy-json.png  

  • Deployment Protection Queries can be extremely versatile. - The only constraint is that query results must be evaluated against a numeric threshold. Otherwise, deployment protection rules can be used to extract very granular data about the state of your application and its systems. For detailed information on queries, refer to Honeycomb Query Builder documentation.

View the Result of Your Actions Workflow Deployments 

After being configured and merged into your repository, Honeycomb deployment protection rules runs when a deployment is requested and waits to receive a pass/fail status before a deployment can occur to the target GitHub environment.

View the results of the Honeycomb deployment protection rules in the GitHub Actions Workflow run. Examples of passing and failing deployment rules appear below.

passing-deployment-rule.png
failing-deployment-rule.png

Whenever a Honeycomb deployment protection rule runs, it is included in the list of all rules protecting deployments in a particular workflow run. The Honeycomb GitHub App appends a permalink to the exact query results used to protect each environmental deployment during the run. Select that permalink to show the data in Honeycomb that allowed your deployment to pass or fail. If your deployment failed, this link provides a jumping off point to begin your investigation.

In order to re-run a deployment protection rule, you must re-run the GitHub Actions workflow stage(s).

Troubleshooting 

GitHub App not Authorized 

If the deployment protection rule fails with “GitHub App not authorized to access Honeycomb team”, uninstall and reinstall the GitHub App.

Changing the Honeycomb Team 

Only one Honeycomb team can be configured during installation of the Honeycomb GitHub App. You cannot change the Honeycomb team where deployment protection rule queries run without re-initializing the authorization flow.

To change the Honeycomb team, you must uninstall and reinstall the GitHub App, and select the desired team during the new installation.

Checking GitHub App Permissions 

You can use Honeycomb to verify the status of the GitHub App installation. In Honeycomb’s left navigation menu, select Account and then select Team settings. Then, select the Integrations tab. Refer to the Honeycomb + GitHub section to determine the installation status.

github-app-integration-page.png

Forcing a Deployment 

GitHub Owners can force a deployment when a deployment protection rule is in a failure state.

Ensure you have the Allow administrators to bypass configured protection rules setting enabled when enabling the Honeycomb Deployment Protection Rule in your GitHub Environment to allow for this.

  admin-config.png  

Maximum Query Limits 

  • Honeycomb deployment protection rules may only query up to the past 24 hours of data
  • No more than one Honeycomb query may be defined per GitHub environment in your deployment