> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Connect Honeycomb and AWS DevOps Agent
> Give AWS DevOps Agent access to Honeycomb telemetry during investigations, and let Honeycomb alerts start new investigations automatically.
AWS DevOps Agent connects to Honeycomb through a one-way integration: Honeycomb sends data and alerts to AWS DevOps Agent, and the agent doesn't send anything back.
With Honeycomb connected, you can:
* **Start investigations automatically**: Configure Honeycomb Triggers and SLO burn alerts to start AWS DevOps Agent investigations through a webhook.
* **Give the agent trace-level context**: AWS DevOps Agent queries Honeycomb telemetry through Honeycomb's remote MCP server while it investigates, pulling in distributed trace data like latency percentiles, error rates, span attributes, and derived columns from your environments.
## How it works
Honeycomb connects to AWS DevOps Agent through two independent mechanisms: an agent space can use either one without the other, though most setups use both.
### Telemetry introspection via MCP
AWS DevOps Agent queries Honeycomb telemetry through a registered MCP server while it investigates.
#### How the connection works
AWS DevOps Agent connects to Honeycomb's remote MCP server as a custom MCP server registration, authenticated with a Management API key scoped to read-only access.
When an investigation needs telemetry context, the agent sends queries through this connection using Honeycomb's standard query model: the same aggregations, filters, and time ranges available in the Honeycomb UI.
Honeycomb returns results over the same connection; the agent never writes to Honeycomb, and its visibility is bounded by the API key's scopes and the Honeycomb team that issued it.
The credential is shared across everyone with access to the agent space; Honeycomb doesn't distinguish between individual AWS users.
Each registration connects to exactly one Honeycomb team; to give an agent space access to more than one team, register each team separately and enable all of them in that agent space.
#### What this gives an investigation
With this connection, an investigation can:
* Break latency down per service and per endpoint, at the percentiles you query on in Honeycomb.
* Separate request latency from long-lived connections, which otherwise appear as extreme duration outliers.
* Attribute tail latency to a specific code path or downstream dependency.
* Cite the traces and span attributes behind a conclusion, so an operator can open the same view in Honeycomb.
### Investigation triggering via webhook
A Honeycomb Trigger or SLO burn alert can start an AWS DevOps Agent investigation automatically, through a webhook.
#### How Honeycomb sends the alert
Honeycomb Triggers and SLO burn alerts use a webhook Recipient to notify AWS DevOps Agent when a condition fires.
Honeycomb authenticates each webhook delivery with a bearer token in the Authorization header, which AWS DevOps Agent generates during webhook creation.
Honeycomb renders a payload template using Go template syntax, populating fields like `priority` and `description` from the firing alert, and sends it to the webhook endpoint.
Honeycomb expects a response within 15 seconds and retries a failed delivery only for a narrow set of conditions: a `408` or `502` response, a `429` or `503` carrying a `Retry-After` header, or a connection-level failure.
#### How AWS DevOps Agent handles it
AWS DevOps Agent validates the payload, then starts a new investigation, deduplicating repeated deliveries for the same alert cycle using the `incidentId` field.
## Before you begin
Make sure you have the following before you connect Honeycomb:
* An agent space.
If you haven't created one, visit AWS's documentation on [Creating an Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html).
* A Honeycomb account with permission to create Management API keys and manage team integrations.
In Honeycomb, both require the Team Owner role.
* The AWS DevOps Agent console open in a region where the service is available.
## Set up telemetry introspection
Set up telemetry introspection so AWS DevOps Agent can query Honeycomb trace data while it investigates, instead of working from AWS telemetry alone.
Honeycomb's MCP server authenticates with a Management API key, a two-part credential: a **Key ID** and a **Key Secret**.
1. In Honeycomb, [create a Management API key](/configure/teams/manage-api-keys/) under **Account** > **Team Settings** > **API Keys**.
When creating your key:
* Name the key something that identifies its purpose, so it's easy to find later (for example, `aws-devops-agent`).
* Grant these permissions:
| Permission | Access | Description |
| - | - | - |
| Model Context Protocol | Read | Lets the MCP server serve query and schema tools to the agent. |
| Environments | Read | Lets the agent enumerate environments and datasets so it can target queries. |
Read access covers everything AWS DevOps Agent needs: it queries Honeycomb telemetry but doesn't modify datasets, Triggers, SLOs, or Boards.
2. Copy both the **Key ID** and the **Key Secret**; you will need them when you register the MCP server in the next step.
The Key Secret displays only once.
If you lose it, delete the key and create a new one.
Register and authorize the Honeycomb MCP server to make it available to every agent space in the account.
You will enable the registration in a specific agent space in the next step.
1. In the AWS DevOps Agent console, select **Capability Providers** from the side navigation.
2. Locate the **MCP Servers** section, and select **Register MCP Server**.
3. On the **Server details** page, enter details:
| Field | Value |
| - | - |
| Server Name | Unique identifier, such as `honeycomb-mcp`. Give each registration a distinct name if you are connecting more than one Honeycomb team. |
| URL | Your Honeycomb MCP endpoint URL, which depends on which Honeycomb region hosts your team. To identify your region, check the URL in your browser when you are logged into Honeycomb and select the corresponding endpoint:- US (default): `https://mcp.honeycomb.io/mcp` (corresponds to `ui.honeycomb.io` UI domain)
- EU: `https://mcp.eu1.honeycomb.io/mcp` (corresponds to `ui.eu1.honeycomb.io` UI domain)
|
| Transport | **Streamable HTTP** |
| Description | Optional server description. |
4. Select **Next**.
5. On the **Authorization flow** page, select **API key**.
6. Select **Next**.
7. On the **Authorization configuration** page, enter the following:
| Field | Value |
| - | - |
| API Key Name | Human-friendly label, such as `honeycomb-mcp` |
| API Key Header | `Authorization` |
| API Key Value | `Bearer :`. Replace the placeholders with your copied Honeycomb API Key ID and Key Secret. Avoid using any wrapping quotation marks or trailing whitespace. Example using placeholder credentials: `Bearer hcmk_01abc23def45ghi67jkl:8mn9opq0rst1uvw2xyz3abc4def5ghi6j` |
8. Review and submit.
Enable the Honeycomb MCP server in each agent space that should use it.
1. In the AWS DevOps Agent console, from the agent spaces page, select an agent space, then select **View details**.
2. Select the **Capabilities** tab.
3. Locate the **MCP Servers** section.
4. Select **Add**.
5. Select the Honeycomb registration you want to enable.
6. Select **Next**.
7. Review and select **Save**.
If you want to give the agent space access to more than one Honeycomb team, repeat this step to enable additional registrations in the same agent space.
Open the agent through **Operator access** on the agent space page and send the following prompt:
```text theme={}
List the available Honeycomb teams and environments.
```
You should receive a response naming your Honeycomb team and its environments, which confirms that the registration and the credential are both correct.
If the agent reports that it can't reach Honeycomb, or that authorization failed, refer to the [Troubleshooting](#troubleshooting) section.
## Set up investigation triggering
Set up investigation triggering so a Honeycomb Trigger or SLO burn alert can start an AWS DevOps Agent investigation automatically, without anyone needing to notice the alert and kick one off manually.
Generate a webhook so Honeycomb has a destination to send alerts to, and a credential to authenticate the request.
1. In the AWS DevOps Agent console, from the agent spaces page, select your agent space, then select **View details**.
2. Select the **Capabilities** tab.
3. In the **Webhook** section, select **Configure**.
4. Select **Generate webhook**.
5. For **Webhook authentication type**, choose **API key**.
6. Copy the webhook endpoint URL and the API key.
The API key can't be retrieved later.
If you lose it, remove the credentials and generate new ones.
To review the general webhook request format and payload schema, visit [Invoking DevOps Agent through Webhook](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-integrations-and-knowledge-invoking-devops-agent-through-webhook.html).
Create a webhook integration in Honeycomb to connect the webhook you generated in the previous step to your Triggers and SLO burn alerts.
To learn more about how webhook recipients work, visit [Send Alerts to Webhooks](/notify/webhooks/).
1. Create the integration.
1. In Honeycomb, select **Account** > **Team Settings** from the navigation menu.
2. Select the **Integrations** view.
3. Locate **Trigger and SLO Recipients** and select **Add Integration**.
4. Enter integration details:
| Field | Value |
| - | - |
| Provider | **Webhook** |
| Name | Human-readable name, such as `aws-devops-agent`. This name appears in the **Recipient** dropdown when you attach it to a Trigger. |
| Webhook URL | Webhook endpoint URL you copied when you generated the webhook. |
| Shared Secret | Leave empty. Honeycomb sends the **Shared Secret** value in the `X-Honeycomb-Webhook-Token` header, which AWS DevOps Agent doesn't use. |
2. Add the authorization header.
1. Select the **Headers** tab.
2. Select **Add header**.
3. Add header details:
| Field | Value |
| - | - |
| Name | `Authorization` |
| Value | `Bearer `. Replace the placeholder with the API key you copied when AWS DevOps Agent generated the webhook. Avoid using any wrapping quotation marks or trailing whitespace. Example using placeholder credentials: `Bearer 01abc23def45ghi67jkl` |
3. Add the payload template.
1. Select the **Payload** tab.
2. Toggle the **Enable** switch on for **Triggers**.
A webhook can be used only with an alert type whose payload is enabled.
3. Replace the contents of the text area with this template:
```json theme={}
{
"eventType": "incident",
"incidentId": "honeycomb-{{ .Alert.InstanceID }}",
"action": "created",
"priority": "HIGH",
"title": {{ printf "[%s] %s" .Environment .Name | toJson }},
"description": {{ toJson .Alert.Description }},
"service": "honeycomb"
}
```
| AWS webhook field | Honeycomb template value | Description |
| - | - | - |
| `eventType` | The literal string `incident` | Required constant. |
| `incidentId` | `honeycomb-{{ .Alert.InstanceID }}` | `.Alert.InstanceID` identifies a single firing of a Trigger. Repeated deliveries carrying the same value deduplicate into one investigation. |
| `action` | The literal string `created` | Map `.Alert.Status` elsewhere; its values are `TRIGGERED` and `OK`, which aren't valid `action` values. |
| `priority` | One of `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, or `MINIMAL` | To send different priorities, create one webhook integration per priority level (for example, `devops-agent-critical` and `devops-agent-high`) and select the matching integration as the recipient on each Trigger. |
| `title` | `{{ printf "[%s] %s" .Environment .Name \| toJson }}` | The Honeycomb environment and Trigger name. Pass through `toJson` because Trigger names can contain quotation marks that would otherwise break the JSON. |
| `description` | `{{ toJson .Alert.Description }}` | Describes which groups or values caused the alert to fire. Pass through `toJson` because `.Alert.Description` contains newlines that would otherwise break the JSON. |
| `timestamp` | Omit | Optional. AWS DevOps Agent records the time it receives the event. |
| `service` | A literal service name | Optional. A static string identifying the source, such as `honeycomb` or your service's name. |
| `data` | Honeycomb context variables | Optional. Everything in `data` passes to the agent as the original event. |
To review the full list of available variables, visit [Custom Webhook Variables](/notify/webhooks/variables/).
To review `toJson` and other transformations, visit [Custom Webhook Functions](/notify/webhooks/functions/).
4. Select **Add**.
To configure SLO burn alerts too, enable the **Budget Rate Burn** and **Exhaustion Time Burn** payload types and adapt the Go template using the variables for those alert types.
Add the webhook you created as a Recipient on each Trigger whose alerts should start an investigation.
1. In Honeycomb, select **Triggers** from the navigation menu.
2. Select an existing Trigger, or select **New Trigger**.
For the full trigger creation flow, including how to define the query and alert condition, visit [Create a Trigger](/notify/triggers/create/).
3. In the **Recipients** section, select **Add Recipient**.
4. From the **Recipient** dropdown, select your webhook integration.
5. Select **Add**, then select **Save Trigger**.
Make sure the Trigger's **Description** field is filled in; this will help the agent as well as your team.
Because the field is delivered in the payload and reads as investigation context, you should state what the Trigger monitors, why it exists, and where to look first.
Select **Test** in the Trigger editor and confirm the following:
* Honeycomb reports that the delivery succeeded.
A `403 Forbidden` response points to a problem with the `Authorization` header.
* An investigation starts in your agent space.
A `200` response without an investigation means the payload was accepted and then failed validation.
The most common causes are a `priority` value that isn't one of the five literal strings, and a free-text field interpolated without `toJson`.
To troubleshoot webhook delivery in general, visit [Invoking DevOps Agent through Webhook](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-integrations-and-knowledge-invoking-devops-agent-through-webhook.html).
## Troubleshooting
If AWS DevOps Agent isn't behaving as expected, check the symptom against the likely cause before opening a support case.
### Telemetry introspection
Most connection problems trace back to how the API key was entered during registration, not the key itself.
#### "Invalid input" error when saving the authorization configuration
This happens when the full header string, rather than just the header name, was entered in the **API Key Header** field.
Enter `Authorization` alone in that field, and move the rest of the value to **API Key Value**.
#### Registration saves, but the agent reports it can't reach Honeycomb
The **API Key Value** is malformed.
Confirm the value is `Bearer :`: both halves present, joined by a colon, with a single space after `Bearer` and no wrapping quotation marks or whitespace.
#### Agent authenticates, but reports no environments
The key is missing the **Environments (Read)** scope, or it belongs to a different Honeycomb team than expected.
Check the key's scopes in Honeycomb, and confirm which team issued it.
#### Agent lists environments, but returns no data for a service
The service isn't sending traces to the environment the key can see, or the query window predates the data.
Confirm in the Honeycomb UI that the dataset holds spans for the time range in question.
#### Requests fail from an EU-hosted team
The registration used the US endpoint instead of the EU one.
Re-register using `https://mcp.eu1.honeycomb.io/mcp`.
To rotate the credential:
1. Create a new Management API key in Honeycomb.
2. Edit the registration's authorization configuration with the new value.
3. Verify with the same prompt used during setup.
4. Delete the old key in Honeycomb.
### Trigger investigations
Most delivery problems either get an explicit rejection, like a `403`, or fail silently with no error at all, so check the response code before assuming the payload itself is wrong.
#### Honeycomb reports "403 Forbidden" on every delivery
The `Authorization` header sent with the webhook doesn't match what AWS DevOps Agent expects.
Delete the header on the **Headers** tab and re-enter it as `Bearer `.
#### "403 Forbidden" persists after you re-enter the header
The webhook credentials were removed or regenerated on the AWS DevOps Agent side, or the endpoint URL doesn't match the webhook that was generated.
Regenerate the credentials in the **Webhook** section of the **Capabilities** tab, then update both the URL and the header in Honeycomb.
Honeycomb doesn't retry `403` responses, so correcting the header only takes effect on the next delivery.
Use **Test** in the Trigger editor to confirm the fix instead of waiting for a real alert to fire.
#### A `200` response comes back, but no investigation starts
Honeycomb delivered the payload successfully, but AWS DevOps Agent rejected it during validation.
This usually means the `priority` value isn't one of the five literal strings (`CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `MINIMAL`), or a free-text field like `title` or `description` was interpolated into a quoted string instead of passed through `toJson`, producing invalid JSON.
Confirm the `priority` value on the **Payload** tab matches one of the five literal strings, and check any free-text field for missing `toJson` wrapping.
`title` and `description` must pass through `toJson` and stay unwrapped by quotation marks.
#### Deliveries fail intermittently with no clear pattern
Honeycomb expects a response within 15 seconds and retries a delivery only for a `408`, `502`, a `429` or `503` carrying a `Retry-After` header, or a connection-level failure such as a refused connection.
If AWS DevOps Agent is slow to acknowledge the request, or returns any other status while under load, the delivery fails without a retry.
Make sure the webhook endpoint acknowledges the request quickly and handles any slower work asynchronously.
#### A repeated alert doesn't start a new investigation
Deliveries within the same alert cycle carry the same `incidentId`, so AWS DevOps Agent treats them as updates to the existing investigation instead of starting a new one.
This is expected behavior; a new investigation starts when the Trigger recovers and fires again.
#### The webhook integration doesn't appear as an option on an SLO burn alert
Only the **Triggers** payload type is enabled on the integration, and a webhook can only be used with alert types whose payload is configured.
Enable the **Budget Rate Burn** or **Exhaustion Time Burn** payload on the **Payload** tab.
To rotate the webhook API key, remove the existing credentials in the **Webhook** section of the **Capabilities** tab, generate new ones, and update the `Authorization` header in Honeycomb.
Requests succeed again once the new header is saved.
#### An investigation starts, then gets cancelled immediately
This usually means your AWS account has hit its monthly investigation limit.
Contact your AWS account team to request a rate limit change.
## Removing the integration
Honeycomb connects at two levels: the agent space level and the account level.
To remove it completely, remove it from all agent spaces first, then unregister it.
The webhook is a separate credential; remove each part you configured.
Delete the integration in Honeycomb to remove it from every Trigger and SLO that used it.
1. In Honeycomb, select **Account** > **Team Settings** from the navigation menu.
2. Select the **Integrations** view.
3. Locate the **Trigger and SLO Recipients** section.
4. Find your webhook integration, then select **Edit**.
5. In the form editor, select **Remove**.
Remove the webhook credentials to stop Honeycomb alerts from triggering new investigations.
1. In the AWS DevOps Agent console, from the agent spaces page, select the agent space, then select **View details**.
2. Select the **Capabilities** tab.
3. In the **Webhook** section, select **Configure**.
4. Choose **Remove**.
To resume triggering investigations, generate new credentials in the **Webhook** section and update the `Authorization` header in Honeycomb.
Remove the registration from an agent space to stop that space from querying Honeycomb, without affecting other agent spaces or the account-level registration.
1. In the AWS DevOps Agent console, from the agent spaces page, select the agent space, then select **View details**.
2. Select the **Capabilities** tab.
3. Locate the **MCP Servers** section.
4. Select your Honeycomb registration, then select **Remove**.
Repeat for every agent space where the registration is enabled.
Once no agent space is using Honeycomb, deregister Honeycomb as an available MCP server for the entire account.
1. In the AWS DevOps Agent console, select **Capability Providers** from the side navigation.
2. Locate the **Currently registered** section.
3. Check that the agent space count is zero.
If it isn't, repeat the previous step in your other agent spaces.
4. Select your Honeycomb registration, then choose **Deregister** from the **Actions** menu.
Deregistering the MCP server doesn't revoke the credential it used, so delete the Management API key in Honeycomb as a separate, final step.
1. In Honeycomb, select **Account** > **Team Settings** from the navigation menu.
2. Select the **API Keys** view.
3. Locate the Management API key you created for this integration and select it.
4. Select **Delete**.
5. In the **Delete Management API Key?** modal, enter the name of the key, then select **Delete**.
## Next steps
Continue setting up AWS DevOps Agent with these resources:
* [AWS DevOps Agent IAM Permissions](https://docs.aws.amazon.com/devopsagent/latest/userguide/aws-devops-agent-security-devops-agent-iam-permissions.html): Review who can register capability providers in your organization.
* [Connecting MCP Servers](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-capabilities-for-aws-devops-agent-connecting-mcp-servers.html): See the full range of ways to connect an MCP server to AWS DevOps Agent.
* [Connecting to Privately Hosted Tools](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-capabilities-for-aws-devops-agent-connecting-to-privately-hosted-tools.html): Configure a private network proxy for AWS DevOps Agent.