Artificial Analysis Endpoint Accuracy Index v1.1 finds 1 in 4 endpoints underperform
Start here
Artificial Analysis released version 1.1 of its Endpoint Accuracy Index, which compares hosted open-weight model endpoints against reference deployments of the same official weights. The firm reports that roughly one in four of more than 100 serverless endpoints scores below the reference deployment of the same open weights. The index reruns each endpoint on BFCL v4-500, HLE-250 and AA-LCR-25. Version 1.1 raises long-context repeats from 10 to 25 and narrows the average 95% confidence interval from plus or minus 4.7 points to plus or minus 2.6 points. It now covers DeepSeek V4.1 Flash 1017, GLM-5.3, GLM-5.3 Flash, Muse Spark Open, Kimi K3 and gpt-oss-120b. In the reported examples, two GLM-5.3 Flash endpoints with output caps near 32,000 tokens scored 22% and 29% raw HLE-250 accuracy versus 55% for the reference, and two Muse Spark Open endpoints that encoded string tool arguments as numbers scored 50% and 51% on BFCL v4-500 versus 59%. Artificial Analysis notes the index uses a fixed evaluation suite and that hosted configurations can change after measurement.
Generated from reports · updated 1 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIArtificial Analysis finds 1 in 4 third-party AI endpoints underperform their advertised models
Artificial Analysis released version 1.1 of its Endpoint Accuracy Index, finding that roughly one in four of more than 100 serverless endpoints scores below the reference deployment of the same open weights. The index reruns each endpoint on BFCL v4-500, HLE-250 and AA-LCR-25 and reports a 95% confidence interval, now narrowed to plus or minus 2.6 points. In the GLM-5.3 Flash results, two endpoints with output caps near 32,000 tokens fell to 22% and 29% raw HLE-250 accuracy versus 55% for the reference, while two Muse Spark Open endpoints that encoded string tool arguments as numbers scored 50% and 51% on BFCL v4-500 versus 59%.
Heat for this event
Not enough continuous observations yet to draw a trend.