AWS Describes How AgentCore Evaluations and DevOps Agent Monitor Production AI Agents

AWS published a blog post explaining how two of its tools, Amazon Bedrock AgentCore Evaluations and AWS DevOps Agent, can be combined to monitor AI agents running in production.

The problem AWS describes

According to AWS, multi-agent systems in production experience failures that traditional monitoring does not catch. The company gives two examples: an agent that cannot invoke its foundation model and returns an empty response because of a missing AWS Identity and Access Management (IAM) permission that does not throw a 500 error, and a supervisor agent with a poorly scoped prompt that begins routing 20 percent of requests to the wrong specialist while infrastructure metrics remain unaffected.

AWS says infrastructure monitoring tools such as Amazon CloudWatch can confirm that a system executed correctly but cannot show whether an agent actually helped a user accomplish their goal. The company writes that an agent can successfully invoke Amazon Bedrock, call every tool without errors, and still return a response that misunderstands what the user needs. AWS also describes a case where a booking agent stops completing reservations even though logs show successful tool executions, because the failure occurred three calls deep in a chain that did not surface an exception.

AWS states these issues compound in multi-agent systems, where a single request triggers a supervisor agent that routes work to multiple specialists, each with its own tools and model calls. AWS says there is typically no fixed execution graph to instrument, and failures can occur at multiple handoff points without predictable propagation through the system.

The system AWS built

To address this, AWS says it built a production airline reservation system with four specialized agents that combines Amazon Bedrock AgentCore Evaluations for continuous agent quality assessment with AWS DevOps Agent for autonomous infrastructure incident investigation.

AWS describes Amazon Bedrock AgentCore as a platform to build, connect, and optimize agents at scale, with any framework or model. Within that platform, AWS says AgentCore Evaluations continuously scores live interactions to catch wrong tool selections, task failures, and quality regressions that infrastructure metrics miss entirely. AWS DevOps Agent, the company says, autonomously traces failures across service boundaries, correlating IAM policies, invocation logs, and orchestration traces without manual investigation. AWS states that together, these two layers are meant to show both whether an agent works correctly and whether the underlying infrastructure supports it.

Key technologies used

AWS lists several services underpinning the system:

• Amazon Bedrock provides API access to foundation models from Anthropic, Meta, Mistral, and Amazon, and powers language understanding in the airline reservation system.
• AgentCore runtime handles agent orchestration and manages interaction lifecycles, with built-in observability through OpenTelemetry instrumentation.
• The Fullstack AgentCore Solution Template (FAST) is referenced as part of the implementation, though the blog post does not fully detail its role in the excerpt provided.

Source: AWS Machine Learning Blog, "Monitoring production agent lifecycle with AWS DevOps Agent and AgentCore Evaluations," published Sept. 11, 2026.