AWS Details Evaluation Framework for AI Agents Built on Bedrock AgentCore
AWS describes an internal evaluation framework built on Amazon Bedrock AgentCore that it says caught agent errors, including a figure mislabeled as $410.08 million, before launch.
한 줄 답
What evaluation framework did AWS build for its AI sales agents using Amazon Bedrock AgentCore?
AWS says it built a continuous evaluation framework, using a capability called AgentCore Evaluations within Amazon Bedrock AgentCore, for a multi-agent system serving AWS Field Sales. The company says pre-launch testing caught errors, including a revenue figure mislabeled as $410.08 million, and that a release mismatch dropped a test pass rate from 78 percent to 38 percent.
핵심 요약
- AWS says its Sales, Marketing, and Global Services (SMGS) Operations team built a multi-agent system, using Amazon Bedrock AgentCore, that serves AWS Field Sales teams worldwide through Amazon Quick.
- AWS says pre-launch evaluation of its Customer Insights Agent found errors including a $410.08 three-month revenue figure mislabeled as '$410.08 Million' and an Annual Recurring Revenue estimate that did not appear in the underlying data.
- AWS says standard monitoring showed the system as healthy during these errors because it only confirmed the agent had responded, not whether the response was correct.
- AWS says that in one pre-production test, a mismatch between data, tool, and prompt updates caused the pass rate to drop from 78 percent to 38 percent.
- AWS describes AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, as the component intended to provide continuous evaluation across the agent's full lifecycle.
AWS published a blog post on October 16, 2026, describing an evaluation framework it built for AI agents in production, centered on a capability it calls AgentCore Evaluations within Amazon Bedrock AgentCore. The post, written by AWS employees Rui Li, Alok Singh, Bharathi Srinivasan, Chandana Srinivas, Mihir Gadgil, and Madhu Samhitha Vangara, describes a system AWS built for its own internal use.
The system being evaluated
AWS says its Sales, Marketing, and Global Services (SMGS) Operations team built a multi-agent system that serves AWS Field Sales teams worldwide. According to the company, sellers and sales leaders can ask business questions in plain English through Amazon Quick, with several specialized sub-agents operating behind that interface:
- The Performance Insights Agent, which AWS says generates natural-language business intelligence narratives from sales data for territory-scoped revenue and performance questions.
- The Customer Insights Agent, which AWS says assembles a 360-degree view of each customer from product usage, support history, contract status, and engagement data.
- The Data Insights Agent, described by AWS as a text-to-SQL agent that lets account teams query structured analytics data without writing SQL.
AWS says answers are scoped to each user's territory and role-level security, and that field teams use the system for account strategy planning and customer meeting preparation, while leadership uses it to power Weekly Business Reviews.
Errors found before launch
AWS says that when it evaluated the Customer Insights Agent before launch, the process caught several errors:
- A customer's three-month revenue of $410.08 on a service was reported as "$410.08 Million."
- A Sales Revenue figure was reported as Sales Revenue excluding Adjustments.
- The agent offered an Annual Recurring Revenue estimate that AWS says appeared nowhere in the underlying data.
- The agent presented its own calculations as sourced facts.
AWS says that throughout these runs, its standard monitoring stayed green, because it only indicated the system had responded, not whether the response was accurate.
Why AWS says risk grows over time
AWS says the risk of these failures increases with every new agent added to the system and with every change to data, prompts, code, or models. The company also says the risk grows even without a new release, because upstream data pipelines refresh weekly. AWS describes a related risk in coordinated releases: when a data change, a tool change, and a paired prompt update are meant to ship together but land apart, the system runs in a mismatched state until all the changes catch up. AWS says that in one pre-production test, this kind of mismatch alone dropped the pass rate from 78 percent to 38 percent.
AgentCore Evaluations
To address these failure modes, AWS says it built a framework intended to show whether agent answers are grounded and correct, and to trace failures back to their root cause in the data, the prompt, or the agent's code. AWS identifies AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, as the component it says powers this framework across the full lifecycle of the agentic system described in the post.
Source: AWS Machine Learning Blog, "Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore," published October 16, 2026.
자주 묻는 질문
- What did AWS announce?
- AWS published a blog post describing an evaluation framework, built on Amazon Bedrock AgentCore, that it uses to test and monitor a multi-agent system serving its own Field Sales organization.
- What errors did AWS say it found before launch?
- AWS says its evaluation process caught a revenue figure of $410.08 mislabeled as '$410.08 Million,' a revenue metric mislabeled as excluding adjustments, an Annual Recurring Revenue estimate not present in the source data, and the agent presenting its own calculations as sourced facts.
- What is AgentCore Evaluations?
- According to AWS, AgentCore Evaluations is a capability of Amazon Bedrock AgentCore that the company built to continuously evaluate agent quality across releases and in production.
- Who built the multi-agent system described in the post?
- AWS says the system was built by its Sales, Marketing, and Global Services (SMGS) Operations team, and it is used by AWS Field Sales teams and leadership worldwide.