Study finds 13 AI models read law more conservatively than their preferred outcomes

Ryan Compton published an independent study testing 13 AI models on 200 split US Supreme Court decisions, giving each model the facts and full opinions and asking two things: whose legal reading was correct, and whether the outcome was good policy. Across the 159 cases where liberal and conservative justices split, the models' legal reading and preferred result diverged 420 times versus 41, almost always favoring the conservative reading of the law while preferring the liberal outcome. The models tested included Claude Opus 5.5, GPT-6.1, Gemini 3.8 Flash, Grok 4.7, Llama 4 Maverick, DeepSeek v4 and Qwen 3.8. The study ran about 3,600 answers through OpenRouter for roughly $150, with one answer per model per case, or two for the 40 cases from a pilot run. The policy question told the models their answer “does not need to match your legal view.” The study notes that no one asked the justices whether each outcome was good for the country, so there is no direct comparison between the models' policy preferences and the justices' own views.

Generated from reports · updated 2 h ago

Not enough continuous observations yet to draw a trend.