AWS launches open-source aws-bench benchmark for testing AI agents on live AWS tasks

AWS has released aws-bench as an open-source benchmark designed to evaluate AI agents on real cloud tasks. Unlike traditional benchmarks that rely on static fixtures, it provisions actual AWS resources in disposable accounts for each scenario. Agents execute tasks in sandboxed containers using scoped credentials tied to those accounts.

Adopting the benchmark requires an AWS organization with credentials that can manage member accounts and organizational units. The setup is currently pinned to the us-east-1 region, so teams operating elsewhere must adapt or accept cross-region latency. Running the benchmark creates persistent resources that may generate charges even when idle. Teams must therefore factor in both permission overhead and potential ongoing cost.

Scoring is performed by an automated verifier, either an LLM judge or a programmatic check against the live AWS state. Because many tasks use an LLM judge, the benchmark shares the same vulnerability that researchers highlighted in other agent scores, namely, that clever prompting can produce high scores without genuine task completion. Leftover state from previous runs can also cause unintended passes or random failures. These factors limit the benchmark’s reliability until further mitigations are added.

The project includes predefined datasets covering observability, compute, data, databases, storage, multi-region EC2, serverless, streaming, IoT, reference architectures and multi-service troubleshooting. Engineering teams can extend these scenarios and tasks to match their own workloads, using the Harbor framework as a base. However, without published baseline results or a leaderboard, users lack a reference point for interpreting scores.