AWS open-sources UAEF, a framework-agnostic evaluation system for AI agents
UAEF: A framework-agnostic system for closed-loop evaluation of AI agents
AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.
The AWS team has open-sourced UAEF, which uses adapters to unify agent trajectories from different frameworks into a single format, then scores them across 7 dimensions using 36 metrics, providing a reusable evaluation layer for comparing agent versions across frameworks.
Source:AWS Machine Learning Blog · aws.amazon.com