AWS Publishes Open-Source Benchmark for Comparing OpenAI Models on Bedrock
AWS released an open-source benchmarking harness and blog post evaluating OpenAI models on Amazon Bedrock against OpenAI API baselines on cost, agent turns, and deliverable quality.
Quick answer
What did AWS announce about comparing OpenAI models on Amazon Bedrock?
AWS published a blog post and open-sourced a benchmarking harness, openai-on-aws/benchmarks-openai, that compares three OpenAI models on Amazon Bedrock against two OpenAI API baseline models on cost per correct answer, multi-turn agent costs, and rubric-graded professional deliverables, rather than on price per token alone.
Key takeaways
- AWS open-sourced a benchmarking harness called openai-on-aws/benchmarks-openai that runs the same OpenAI Responses API code path against models on Amazon Bedrock and on the OpenAI API.
- The harness evaluates three OpenAI models on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol) against two cost-efficient OpenAI API models (gpt-5.4-mini, gpt-5.4-nano) that AWS describes as common cost-optimized baselines rather than direct generational peers.
- AWS says the evaluation looks at cost per correct answer, the cost of multi-turn agent trajectories, and performance on rubric-graded professional deliverables, instead of relying on price-per-token comparisons alone.
- AWS states that models on Amazon Bedrock were run with reasoning disabled while OpenAI API models ran at default settings, and describes the results as reflecting practical deployment configurations rather than a controlled measure of intrinsic model capability.
- Grading in the harness combines deterministic checks with an LLM judge model, gpt-5.5, using frozen prompts, and AWS reports sample sizes ranging from 48 to 198 items per benchmark.
AWS published a blog post on its Artificial Intelligence blog describing an open-source benchmarking harness for comparing OpenAI models available on Amazon Bedrock against OpenAI models accessed directly through the OpenAI API. The post is credited to AWS authors Nick McCarthy, Sharadha Kandasubramanian, Sudeesh Sasidharan, and Saurabh Trikande.
AWS says organizations typically compare models using dollars per million tokens, a figure it calls the number on every pricing page. According to the post, that figure misses factors that affect real costs: how often a model produces a correct answer, how many tokens it consumes to get there, and, for agentic workloads, how many conversational turns it takes, since each turn resends the growing conversation history.
The harness and the models tested
AWS says it built and open-sourced a harness, published as openai-on-aws/benchmarks-openai, that runs one identical code path, the OpenAI Responses API, against both Amazon Bedrock and the OpenAI API by switching the backend and model ID while keeping evaluation logic constant.
The post states the harness evaluates five models:
- Three OpenAI models on Amazon Bedrock: gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol
- Two OpenAI API models: gpt-5.4-mini and gpt-5.4-nano
AWS describes the two OpenAI API models as cost-optimized baselines that many teams already use, not as direct generational counterparts to the Bedrock models, framing the comparison around the question of whether a newer Bedrock model is worth moving to from mini or nano.
What the evaluation covers
According to AWS, the harness addresses three questions: what a correct answer costs rather than what a token costs, what multi-turn agent trajectories cost given that turn count can dominate a bill, and whether a model can produce work a professional would accept, tested against real occupational deliverables rather than quiz-style questions.
AWS says the evaluation measures single-call accuracy and cost on AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro, alongside multi-turn agent trajectories on live web-research tasks and rubric-graded professional deliverables. Grading combines deterministic checks with an LLM judge, gpt-5.5, which AWS notes is not one of the models being evaluated, using frozen prompts whose hashes are recorded in each result file.
AWS states that every run of the harness writes a timestamped result file in JSON format, and that every number and chart in the post was generated from those files at build time. The source material provided does not include the specific accuracy, cost, or benchmark results the harness produced.
Source: AWS Machine Learning Blog, "Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload," published September 11, 2026.
Frequently asked questions
- What is openai-on-aws/benchmarks-openai?
- It is an open-source benchmarking harness that AWS says runs an identical OpenAI Responses API code path against OpenAI models hosted on Amazon Bedrock and against the OpenAI API, so users can reproduce the comparison on their own workloads.
- Which models does the AWS benchmark compare?
- AWS compares three OpenAI models on Amazon Bedrock — gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol — against two OpenAI API models, gpt-5.4-mini and gpt-5.4-nano, which it calls widely used cost-efficient baselines.
- What does AWS say it measures besides price per token?
- AWS says it measures the cost of a correct answer on benchmarks including AIME, GPQA Diamond, and MMLU-Pro, the cost of multi-turn agent trajectories on live web-research tasks, and performance on rubric-graded professional deliverables.
- Did AWS run the compared models under identical settings?
- No. AWS states the Amazon Bedrock models ran with reasoning disabled while the OpenAI API baseline models ran at their default settings, and it describes the comparison as reflecting practical deployment configurations rather than a controlled test of intrinsic model capability.