AWS Details Evaluation Framework for AI Agents Built on Bedrock AgentCore

AWS published a blog post on October 16, 2026, describing an evaluation framework it built for AI agents in production, centered on a capability it calls AgentCore Evaluations within Amazon Bedrock AgentCore. The post, written by AWS employees Rui Li, Alok Singh, Bharathi Srinivasan, Chandana Srinivas, Mihir Gadgil, and Madhu Samhitha Vangara, describes a system AWS built for its own internal use.

The system being evaluated

AWS says its Sales, Marketing, and Global Services (SMGS) Operations team built a multi-agent system that serves AWS Field Sales teams worldwide. According to the company, sellers and sales leaders can ask business questions in plain English through Amazon Quick, with several specialized sub-agents operating behind that interface:

• The Performance Insights Agent, which AWS says generates natural-language business intelligence narratives from sales data for territory-scoped revenue and performance questions.
• The Customer Insights Agent, which AWS says assembles a 360-degree view of each customer from product usage, support history, contract status, and engagement data.
• The Data Insights Agent, described by AWS as a text-to-SQL agent that lets account teams query structured analytics data without writing SQL.

AWS says answers are scoped to each user's territory and role-level security, and that field teams use the system for account strategy planning and customer meeting preparation, while leadership uses it to power Weekly Business Reviews.

Errors found before launch

AWS says that when it evaluated the Customer Insights Agent before launch, the process caught several errors:

• A customer's three-month revenue of $410.08 on a service was reported as "$410.08 Million."
• A Sales Revenue figure was reported as Sales Revenue excluding Adjustments.
• The agent offered an Annual Recurring Revenue estimate that AWS says appeared nowhere in the underlying data.
• The agent presented its own calculations as sourced facts.

AWS says that throughout these runs, its standard monitoring stayed green, because it only indicated the system had responded, not whether the response was accurate.

Why AWS says risk grows over time

AWS says the risk of these failures increases with every new agent added to the system and with every change to data, prompts, code, or models. The company also says the risk grows even without a new release, because upstream data pipelines refresh weekly. AWS describes a related risk in coordinated releases: when a data change, a tool change, and a paired prompt update are meant to ship together but land apart, the system runs in a mismatched state until all the changes catch up. AWS says that in one pre-production test, this kind of mismatch alone dropped the pass rate from 78 percent to 38 percent.

AgentCore Evaluations

To address these failure modes, AWS says it built a framework intended to show whether agent answers are grounded and correct, and to trace failures back to their root cause in the data, the prompt, or the agent's code. AWS identifies AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, as the component it says powers this framework across the full lifecycle of the agentic system described in the post.

Source: AWS Machine Learning Blog, "Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore," published October 16, 2026.