AWS open-sources UAEF agent evaluation framework
Start here
AWS has open-sourced UAEF (Universal Agent Evaluation Framework) under the Apache 2.0 license in the awslabs/uaef GitHub repository. The framework-agnostic system uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into a single canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Running the library alone creates nothing; persisting a run writes experiment metadata to Amazon DynamoDB and full results to Amazon S3, letting teams compare agent versions against a baseline and flag regressions at a default 5 percent threshold, with statistical reports and LLM-written insights. AWS says 12 organizations in telecom, healthcare and financial services used UAEF before release. The DynamoDB tables, S3 buckets and Cognito user pool are created with a retain policy, so deleting the stack leaves them and their contents in place.
Generated from reports · updated 20 h ago
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogPicksAWS open-sources UAEF, a framework-agnostic evaluation system for AI agents
AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.
Heat for this event
Not enough continuous observations yet to draw a trend.