AI Ecosystem is only available in Early Access.
Contact your Honeycomb account team for more information.
Overview
AI Ecosystem brings together the views you use to monitor and investigate your AI agents: fleet-wide performance, estimated LLM cost, and the conversations behind them. When you run many agents, finding the conversation that explains a failure spike or a jump in spend is often the hardest part of an investigation. AI Ecosystem works from the agent telemetry you already send, so the questions it answers and the numbers it shows come from your own conversations. With AI Ecosystem, you can:- Check fleet health: Review conversation volume, active agents and tools, failure rate, and token usage without writing a query.
- Track estimated spend: Break down LLM cost by agent, model, or both.
- Decide where to look first: Compare every agent in your fleet by its key health metrics.
- Drill down to a conversation: Open the conversation behind an aggregate number in the Agent Timeline.
How AI Ecosystem works
AI Ecosystem answers questions at the fleet and conversation levels, tracks a core set of signals across your agents, and estimates what your LLM calls cost.Questions AI Ecosystem answers
AI Ecosystem helps you answer questions at two levels: across your whole agent fleet, and within a single conversation.- Across your fleet: Which agents are failing or slowing down? Is a problem isolated to one agent, or spread across many? Which agents and models drive your LLM spend?
- Within a conversation: What happened, step by step? Which LLM call or tool call failed, and when?
How trends connect to conversations
AI Ecosystem builds its charts and tables from the same GenAI spans that power the Agent Timeline. Because the aggregates and the conversations are the same data, each number connects to the conversations that produced it. You can find the conversation behind a trend without writing a query or correlating data by hand. To learn how to instrument your agents to emit spans with GenAI attributes, visit Instrumenting AI Agents.Measured signals
AI Ecosystem tracks these kinds of signals across your agent fleet:- Activity: Conversation volume, active agents, active tools, and token usage.
- Reliability: Failures and failure rate.
- Responsiveness: GenAI span latency and time to first token, which measures how quickly your models start responding.
- Cost: Estimated LLM spend, in total and per conversation, broken down by agent, model, and token type.
How LLM cost is estimated
Honeycomb calculates cost when your spans arrive, using the model and token counts on each LLM call and a public price catalog. Cost accrues on the LLM calls themselves; agent, tool, and workflow spans aren’t priced directly, because the LLM calls beneath them already carry the cost.Cost values are estimates.
They don’t reflect negotiated rates, discounts, free tiers, or credits, so use them to understand what drives spend rather than to reconcile a provider invoice.
Honeycomb currently prices spans from the
openai, anthropic, and aws.bedrock providers.
To learn more, visit How Honeycomb Calculates LLM Costs.