Vals AI and NextLadder Release Access to Justice Benchmark

Vals AI and NextLadder Ventures released the Access to Justice Benchmark on November 10, testing 23 AI models on 252 simulated legal-aid conversations. No model reached the 80% threshold the benchmark sets for a reliable first point of contact; the top scorer, Alibaba's Qwen 3.8 Max, reached 76.4%. Models met 78–87% of criteria for general legal explanations but only 30–50% for identifying specific local resources, and 21 of 23 fell back on national directories in at least 70% of final answers. Free-tier chatbots averaged 69.7% versus 70.4% for premium ones; only xAI showed a clear premium edge, with Grok 4.7 beating Grok 4.6 by about 2.3–2.4 points. The published results use only the 252 public scenarios from a 504-scenario dataset. Models were run through APIs, which approximates but does not reproduce consumer chatbot behavior, and the free/premium tier assignments are Vals AI's estimates rather than confirmed provider mappings. Content-filter refusals counted as zero, which Vals AI says explains Muse Spark 1.3's 41.5% on criminal and records questions.

Generated from reports · updated 21 min ago

Not enough continuous observations yet to draw a trend.