Skip to content
Google AI · DEV· Patrick Kelly·· 3 h agoAI score58

Testing Gemini Decision 1: 800 Scenarios Show Narrow Questions Beat Broad Ones

Exploring Decision Models, Part 1: Putting Gemini Decision 1 to the Test

AI summary

An independent test of Google's Gemini Decision 1 across 800 scenarios in 24 categories found that splitting one broad routing question into four or five narrow ones raised accuracy from 0.72 to 0.88. Customer support routing hit nearly 100% accuracy at 230ms per request and $0.34 per 10,000 requests, prompt injection detection reached 99.7%, and memory-update filtering reached 88%, while predicting user pushback scored 46% and splitting code files for retrieval was beaten by plain deterministic code. The model returns true/false, a score, or a category in one pass, costs $0.10 per million input tokens with no output token cost, and handled 100 questions in about 250ms versus 180ms for one.

Source:Google AI · DEV · dev.to