Skip to content
Only Tutorials#Reasoning
  1. vLLM Blog72

    vLLM serves DeepSeek-V4.1-Flash on Vera Rubin NVL72 at 58x GB300 NVL72 throughput

    vLLM reports serving DeepSeek-V4.1-Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7M tokens/s per GPU with 387 tokens/s per user, 58x the GB300 NVL72, peaking at 2.52M tokens/s and 1,810 tokens/s for a single user. The gains come from kernels rebuilt for Rubin's split memory domains and FP4 tensor cores, a specialized TP8 decode path, and rack-wide KV and Engram pools shared over NVLink. With prefixes pooled across the rack and Dynamo workload-aware routing, the prefix hit rate holds at 96.5-97% even at high concurrency.

    Why it matters: vLLM's third DeepSeek-V4.1-Flash post details how Rubin-specific kernels and rack-wide memory pooling lift throughput 58x over GB300 NVL72, offering a concrete reference for large-scale serving optimization.

  1. Hugging Face Blog60

    Hugging Face explains decision models and their support in Transformers and llama.cpp

    Hugging Face published an explainer on decision models, which take a state plus typed questions and return probabilities over predefined answers instead of generating text. The post details the choice, score and noul primitives, the confidence formula, and new support through Transformers' DecisionPipeline and AutoModelForDecision, a shared /v1/systemone endpoint in Transformers and llama.cpp, a Hub tag for discovery, and fine-tuning with the Trainer API. It notes that the same endpoint also works with compatible autoregressive LLMs by scoring answer logits in a batched forward pass.

    Why it matters: Hugging Face explains how decision models turn classification into a typed, non-generative interface and shows how to serve and fine-tune them with Transformers and llama.cpp.

  1. Modal Blog60

    Modal engineers reverse-engineer how Flash Attention 4 works

    Modal engineers read the released Flash Attention 4 source code and reverse-engineered how the Blackwell-optimized attention kernel works, since no official technical report exists yet. The write-up traces a tile through the kernel's warp-specialized asynchronous pipeline and details two math tricks: a cubic-polynomial software exponential that can replace SFU-based exponentiation, and a smarter online softmax rescaling rule that Tri Dao said cuts corrections by a factor of 10. FA4 is reported to be roughly 20% faster than Nvidia's closed-source cudnn attention kernels, and a backwards version was not yet available at the time of writing.

    Why it matters: Modal engineers read the released Flash Attention 4 source and explain its warp-specialized pipeline and two math tricks, offering a transferable view of where CUDA kernel design is heading.

  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

  1. Cursor Changelog62

    Cursor analyzes Llama-2-70B inference cost and latency against GPT-3.5

    Cursor measured Llama-2-70B inference on 2 80-GB A100 GPUs and found it over 3x cheaper than GPT-3.5 for prompt tokens but far more expensive for generated tokens at small batch sizes. Prompt processing costs about $0.00042 per 1K tokens versus GPT-3.5's $0.0015, while generation reaches $0.066 per 1K tokens at batch size 1 and only approaches GPT-3.5 pricing at large batches that push time to first token to several seconds. Cursor recommends open-source models for prompt-heavy tasks like classification and reranking, and closed-source APIs for generation-heavy workloads.

    Why it matters: Cursor's measured cost and latency data shows when self-hosting Llama-2-70B beats GPT-3.5 and when it does not, offering a reusable framework for open-source model serving decisions.

You are all caught up