OpenRouter published a guide for benchmarking LLMs on your own application's inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats, while DeepSeek had the lowest response cost at 157/180 passes. Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
vLLM reports serving DeepSeek-V4.1-Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7M tokens/s per GPU with 387 tokens/s per user, 58x the GB300 NVL72, peaking at 2.52M tokens/s and 1,810 tokens/s for a single user. The gains come from kernels rebuilt for Rubin's split memory domains and FP4 tensor cores, a specialized TP8 decode path, and rack-wide KV and Engram pools shared over NVLink. With prefixes pooled across the rack and Dynamo workload-aware routing, the prefix hit rate holds at 96.5-97% even at high concurrency.
Why it matters: vLLM's third DeepSeek-V4.1-Flash post details how Rubin-specific kernels and rack-wide memory pooling lift throughput 58x over GB300 NVL72, offering a concrete reference for large-scale serving optimization.
NVIDIA Research and Inferact's REINFORCE Zero paper shows that aligning rollout and trainer kernels lets unclipped token-level importance sampling correct the remaining weight lag, training stably for 20,000 updates at policy lag 64 and reaching 2.40x the training-step throughput of synchronous training with aligned kernels. With standard kernels, the median rollout-to-learner KL rises 83.33x across the run and online AIME24 pass@1 falls from 34.27% to 13.96%, while a sequence gate stabilizes training but rejects 24.48% of sequences by the last 1,000 updates. The zero-mismatch setup now covers DeepSeek-V4.1-Flash, Kimi K3.1 on release day, and DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X, with code in Inferact/zero-mismatch-rl and NVIDIA-NeMo/labs-molt.
Why it matters: NVIDIA Research and Inferact split the async RL rollout-learner gap into weight lag and kernel mismatch, showing that aligning kernels lets unclipped importance sampling hold stable training at policy lag 64.
DeepSeek V3 processes 47% more tokens per second than R1 at $0.28 per million output tokens, while R1 costs $2.19 per million and runs internal reasoning loops for harder tasks. R1 scores 97.3% on MATH-500 and 79.8% on AIME 2024 versus V3's 90.2% and 39.2%. Most teams route 80-90% of simple queries to V3 and send complex reasoning work to R1.