Skip to content
Hot eventDeveloping

Independent test of Gemini Decision 1 across 800 scenarios

1 reports1 reporting sourcesUpdated 3 h ago

Start here

AI digest

Patrick Kelly published an independent hands-on evaluation of Google's Gemini Decision 1 across 800 scenarios in 24 categories, built from his own coding sessions and a public dataset. He reports that splitting one broad routing question into four or five narrow ones raised accuracy from 0.72 to 0.88; customer support routing reached nearly 100% at 230ms per request and $0.34 per 10,000 requests, prompt injection detection 99.7%, and memory-update filtering 88%. Predicting user pushback scored 46%, and splitting code files for retrieval was beaten by plain deterministic code. The model returns true/false, a score, or a category in one pass. Kelly says it costs $0.10 per million input tokens with no output token cost, plus up to 10,000 free requests per day on Google's free tier, and that up to 1,000 questions can be asked in a single request with little added latency per question; 100 questions took about 250ms versus 180ms for one. He notes the tests are not exhaustive.

Generated from reports · updated 3 h ago

Report timeline

Follow the reports to see the event from different sides.

November 20
  1. Google AI · DEV
    Testing Gemini Decision 1: 800 Scenarios Show Narrow Questions Beat Broad Ones

    An independent test of Google's Gemini Decision 1 across 800 scenarios in 24 categories found that splitting one broad routing question into four or five narrow ones raised accuracy from 0.72 to 0.88. Customer support routing hit nearly 100% accuracy at 230ms per request and $0.34 per 10,000 requests, prompt injection detection reached 99.7%, and memory-update filtering reached 88%, while predicting user pushback scored 46% and splitting code files for retrieval was beaten by plain deterministic code. The model returns true/false, a score, or a category in one pass, costs $0.10 per million input tokens with no output token cost, and handled 100 questions in about 250ms versus 180ms for one.

Heat for this event

Current, comparable range
9
Peak, comparable range
10November 20 01:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.