OpenRouter benchmarks 11 rerankers on 274-question RAG test
Start here
OpenRouter published a tutorial and benchmark comparing 11 rerankers for retrieval-augmented generation (RAG) pipelines. On a 274-question synthetic benchmark, plain vector search with google/gemini-embedding-2 placed the exact right chunk first 81.0% of the time; reranking the top 30 candidates with voyageai/rerank-3-lite raised that to 97.1% for about $0.10 per 1,000 queries. OpenRouter recommends retrieving about 30 chunks, reranking them, and passing only the top 3 to the LLM, which in its test raised answer accuracy to 99.3% and cut answering cost from $0.22 to $0.16 per 1,000 questions. OpenRouter cautions that the test used a small synthetic dataset, so real-world gains depend on the user's documents. It also reports two failure modes: omitting the input_type parameter cost 7 to 12 points of precision@1, and cohere/rerank-v3.5 silently ignores content beyond its 4,096-token limit.
Generated from reports · updated 10 h ago
Report timeline
Follow the reports to see the event from different sides.
- OpenRouter AnnouncementsOpenRouter tests 11 rerankers on a 274-question RAG benchmark
OpenRouter benchmarked 11 rerankers on a 274-question synthetic RAG dataset and found that reranking the top 30 vector-search candidates with voyageai/rerank-3-lite raised precision@1 from 81.0% to 97.1% at about $0.10 per 1,000 queries. The guide recommends retrieving roughly 30 chunks, reranking them, and sending only the top 3 to the LLM, which in its test lifted answer accuracy to 99.3% while cutting answering cost from $0.22 to $0.16 per 1,000 questions. It also flags failure modes: omitting the input_type parameter cost 7 to 12 points of precision@1, and cohere/rerank-v3.5 silently ignores content beyond its 4,096-token limit.
Heat for this event
Not enough continuous observations yet to draw a trend.