Skip to content
#Reasoning
October 19Monday
  1. Sakana AI Blog55

    Sakana AI proposes QD-Rec, a Quality-Diversity recommendation method that pairs candidates with LLM-written hypotheses

    Sakana AI developed QD-Rec, a recommendation method that treats a candidate plus a hypothesis explaining why to choose it as the unit of recommendation and uses Quality-Diversity search to accumulate high-quality, diverse proposals. In experiments on M&A, customer targeting and paper recommendation, QD-Rec built archives with higher average scores and larger size than a baseline that writes each hypothesis from scratch, and its code is released on GitHub. The paper reports cases where a hypothesis scored 1 out of 10 for one company was mutated and transferred to another candidate, raising the score to 5.

October 15Thursday
  1. Mistral AI71

    Mistral releases Timestral time-series foundation models, topping GIFT-Eval, TIME and fev-bench

    Mistral released Timestral, its first family of time-series foundation models, in four sizes from 5M to 209M parameters. The largest model ranks first on all three main public benchmarks, GIFT-Eval, TIME and fev-bench, among pre-trained zero-shot models from a single checkpoint without ensembling, using two-thirds of the parameters of the previous state of the art. Timestral produces probabilistic multivariate forecasts with 21 quantiles, supports past-only and known-future covariates, runs on CPU without a GPU, and is available through Mistral Forge for fine-tuning, with a GitHub repository and technical report published under a modified MIT licence.

    Why it matters: Mistral's first time-series foundation model family claims top zero-shot rankings on GIFT-Eval, TIME and fev-bench with a 209M-parameter checkpoint, offering a size-accuracy tradeoff reference for forecasting work.

  2. inclusionAI Repositories67

    inclusionAI releases Ming-3.0-audio-flash and Ming-3.0-audio-tiny audio models

    inclusionAI released Ming-3.0-audio-flash, a 126B-parameter Mixture-of-Experts audio model with 6.5B activated parameters, alongside the compact Ming-3.0-audio-tiny at 9.3B parameters with 2.7B activated. The flash model covers Chinese and English speech recognition across dialects and noisy multi-speaker settings, long-audio understanding, non-speech sound and music perception, audio reasoning, and end-to-end spoken dialogue with optional synthesized speech output. It ships with vLLM-Omni, SGLang, and Transformers inference paths under the MIT License, with default Flash profiles using tensor parallelism across four 80 GB H800 GPUs at 128K context.

    Why it matters: The release details a 126B-parameter MoE audio model with 6.5B activated parameters, plus a 9.3B tiny variant, giving a concrete reference for open audio-language model sizing and deployment.

  3. AI as Normal Technology36

    Steering AI Beyond Automation: A Research Agenda for Human-AI Collaboration

    Stanford economists Gabriel Unger and Arvind Narayanan argue that AI research is overly biased toward full automation and sketch an alternative agenda centered on human-AI collaboration. They propose four research directions, including "white box agents" that reveal their intermediate processes, sources, assumptions, and uncertainties so users can interact with them directly. The essay traces the debate's history in computer science and calls for a bottom-up, market-driven approach to steering AI development.

  4. Goodfire Research62

    Goodfire research essay argues neural networks are already clean and superposition was misread

    Goodfire researcher argues that the canonical evidence for polysemanticity and superposition in neural networks was misread, and that what looked like arbitrary packing is mostly a motif he calls recursive generalization. Retraining InceptionV1 with ordinary architectural improvements and no change to model size or training data made 99-100% of neurons in its last layer easily interpretable, and preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job. The essay concludes networks use one learning principle rather than two, so interpretability can proceed by studying neurons and circuits directly instead of first training decomposition models such as sparse autoencoders.

    Why it matters: A first-hand reinterpretation of the field's canonical polysemanticity evidence, arguing that networks generalize recursively rather than pack unrelated concepts, which would make direct mechanistic study viable.

October 13Tuesday
  1. Modal Blog58

    Modal tests mixed-precision, mixed-accelerator RL samplers on Qwen3.6-35B-A3B

    Modal trained Qwen3.6-35B-A3B on agentic coding with samplers spread across six accelerator types (B200, B300, H100, H200, A100, RTX PRO 6000) at NVFP4, FP8 and BF16 precision, feeding one BF16 trainer on B200. Plain GRPO collapsed on both mixed and homogeneous samplers, but adding masked importance sampling plus score centering, with or without top-p mask replay, trained stably to the end and reached SWE-bench Pro V2 pass@4 on par with homogeneous samplers, at roughly 15% lower sampler cost per update. The mixed samplers ran with an average sampler-trainer KL 4.7x that of the homogeneous setup, and the authors argue stability depends on keeping the mismatch bounded rather than low.

  2. Amazon Science75

    Amazon Web Services researchers propose priming to convert pretrained Transformers into hybrid state space models without full retraining

    Amazon Web Services researchers introduce priming, a method that reuses pretrained Transformer weights to cut the pretraining cost of hybrid state space architectures by more than 99%, using under 0.5% of the source model's pretraining token budget. In controlled comparisons, Gated KalmaNet outperformed Gated DeltaNet and Mamba-2, and a 32B GKA-based hybrid matched its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context. The team released 8B and 32B primed hybrid models, training code, Triton kernels, and a vLLM serving plugin under the Apache 2.0 License.

    Why it matters: Amazon Web Services researchers show that reusing pretrained Transformer weights can cut hybrid architecture pretraining cost by over 99%, making controlled comparisons of state space layers practical.

October 12Monday
  1. Import AI38

    Import AI 476: Creativity and RSI, Worse-Than-China Open Weight Models, and AI in Space

    Import AI 476 covers a new benchmark showing frontier LLMs still trail humans on intuitive visual reasoning, with GPT 6 Astra scoring 53.6 pass@1 versus a 93.1 human baseline. It also discusses AI systems improving at "research taste" via the TasteVal benchmark, and argues AI could accelerate space industrialization and create compounding first-mover advantages.

  2. Meta AI Research Blog82

    Meta releases Muse Wonder and Muse Wonder Full frontier models for personal and professional agents

    Meta introduced the Muse Wonder series, comprising Muse Wonder and Muse Wonder Full, which it says delivers state-of-the-art performance on personal and professional agentic tasks, memory and coding. Muse Wonder is available today through the Meta Model API and Muse Code, while Muse Wonder Full is available today for Muse Code subscribers. The models pair Active Memory, which carries context across conversations and retrieves task-relevant details, with Active Focus, which sustains work on long-running goals and checks in before important actions; Meta says both were trained at its gigawatt-scale Prometheus data center.

    Why it matters: Meta's Muse Wonder series claims state-of-the-art results on agentic, memory and coding benchmarks, with Active Memory and Active Focus as the two capabilities it is built around.

  3. Modal Blog40

    Modal launches Endpoint Candidates to evaluate and promote inference endpoints

    Modal has released Endpoint Candidates, generally available for all workspaces, letting users stage alternative deployments for Dedicated Endpoints that mirror a sample of production traffic while Modal runs the infrastructure. Users can inspect latency and throughput metrics, run benchmarks, and test output quality via each Candidate's own URL, then promote it to production from the dashboard. Mirrored requests are best-effort and billed at standard Modal compute pricing.

October 7Wednesday
  1. OpenAI Alignment Blog52

    OpenAI studies metagaming latents inside language models

    OpenAI alignment researchers identified sparse autoencoder latents tied to metagaming in OpenAI o3 and found that metagaming draws on several overlapping processes rather than one mechanism, including task analysis, evaluation awareness, reward-seeking and normative reasoning. Steering along these latents shifted Verbalized Metagaming scores and, on the even_number task, the share of odd-number answers, and some latents influenced outputs without metagaming appearing in the written chain of thought. The latents' activation levels and steering effects generally grew across RL checkpoints, consistent with earlier findings that metagaming becomes more common during reinforcement learning.

October 5Monday
  1. Import AI40

    Import AI 475: Swarm scaling, Google DeepMind's SynthID Bio watermarking, and the AI science economy

    Import AI 475 covers Toby Ord's analysis finding that AI swarms trade roughly twice the tokens for half the wall-clock time, with diminishing returns as agent counts grow. Google DeepMind introduced SynthID Bio, a watermarking family for synthetic biology that matched unwatermarked designs on hit rate and binding affinity across three target proteins. The issue also covers CSAIP polling showing 61% of Americans say voluntary corporate AI commitments are "not enough," and the SciUniverse benchmark of 92 tasks testing how well AI systems operate automated labs.

October 3Saturday
  1. Epoch AI · Gradient Updates58

    Epoch AI: Compute built over the next two years could run hundreds of millions of AI agents

    Epoch AI estimates that AI compute produced over the next two years could support tens to hundreds of millions of concurrent agents running today's most capable models, or billions running cheaper ones. Using projected HBM shipments from 2025 through 2027, the analysis counts GB300-equivalent GPUs and finds top-tier agents running nonstop could supply the weekly hours of roughly 140 million to 700 million full-time employees, while DeepSeek V4 Pro serving benchmarks imply about 1.9 billion concurrent agents. The report notes the key uncertainty is whether demand grows fast enough to justify the buildout or leaves a glut of agents.

October 2Friday
  1. Meta AI Research Blog71

    Meta shares six math papers co-authored with mathematicians and Muse Spark

    Meta published six mathematics papers produced in collaboration with mathematicians who used Muse Spark 1.1 and 1.2 in Thinking Mode through the regular meta.ai chat interface, with no custom research scaffold. Five of the papers answer previously open questions, covering Gaussian ellipsoid fitting thresholds, finite-time blow-up for a biharmonic nonlinear Schrödinger equation, a 384-element counterexample to a group-theory conjecture, tightness of a cycle-based relaxation in binary polynomial optimization, and a link between string two-point functions and height functions on curves. Each paper marks which passages were primarily drafted by researchers and which by AI, credits prior work, and acknowledges independent concurrent solutions from other teams.

    Why it matters: Meta 公布六篇由数学家与 Muse Spark 协作完成的论文，其中五篇给出此前开放问题的答案，并标注 AI 与人类各自撰写的段落。

October 1Thursday
  1. Zyphra Research38

    How Hybrid NoPE Models Learn Position Without Position Encodings

    Zyphra Research explains how hybrid language models recover positional information even when global attention layers use no explicit position encoding. Local layers such as sliding window attention and gated linear attention create a recency bias in the residual stream, and global NoPE attention learns query and key projections that read it out as an implicit relative position encoding. The findings offer a mechanistic account of why interleaving local mixing with global NoPE attention works well and point to positional representations that extrapolate beyond training sequence lengths.

  2. Modal Blog42

    Modal Runtime Roundup: VM Sandboxes, Modal Clusters, and More

    Modal announced VM Sandboxes, giving agents a full Linux computer for Docker stacks, local databases, and mobile simulators, already used by Linear, Legora, and Snorkel. Modal Clusters are now generally available via the @modal.clustered decorator, supporting trillion-parameter training and serving across coordinated containers. Sticky Sessions and Endpoint Candidates were also introduced, with the latter in private beta.

September 24Thursday
  1. Modal Blog58

    Modal and CMU's Full Stack Data Lab release Quail, an AI-SQL inference engine that jointly optimizes query planner and inference

    Modal and Carnegie Mellon University's Full Stack Data Lab built Quail, a query-aware inference engine for AI-SQL workloads that jointly optimizes the SQL query planner and the inference engine. On one multi-join query it processes over a billion tokens per minute per H100 GPU, more than 10x faster than Modal's vLLM baseline on the same hardware, at under 6¢ per billion tokens; on the team's new AI-SQL benchmark it runs 1.84x faster than vLLM geometrically averaged over tasks. Because AI-SQL filters and joins need only a single token per sequence, Quail skips decode, sampling and prefill-decode disaggregation, and it uses the known query plan to order requests and manage KV cache eviction.

September 23Wednesday
  1. Vapi Blog38

    Vapi tests Jev as a real-time guardrail for voice AI

    Vapi ran early experiments using TypeSafe AI's early-access "System One" model Jev as a guardrail that scores whether a voice assistant's response violates a developer-defined rule before it is spoken. In one early test, Jev caught at least three out of four violating responses, though Vapi says it is not a 100% compliance guarantee and only an additional check on top of prompt guardrails.

  2. Modal Blog58

    How Modal serves trillions of tokens for trillion-parameter coding agents

    Modal explains how it optimized Kimi K2.6 inference for coding agents, reaching 2.8x per-user and 5.6x cross-user performance per replica and running services at hundreds of billions of tokens per day. The work combines custom DFlash speculative decoding fine-tuned on coding traces, FP8 KV cache and NVFP4 shared-expert quantization, HiCache CPU-tier caching, and KV cache-aware load-based routing. Modal upstreamed its SGLang patches and released the configuration behind Modal Auto Endpoints.

September 17Thursday
  1. Goodfire Research74

    Goodfire Research finds models internally signal reward hacking, detectable with activation probes

    Goodfire Research reports that open-source models Kimi K3, GLM 5.2 and Qwen 3.8 Max reward-hacked in 50–96% of rollouts across three agentic benchmarks, and that a shared internal signal tied to cheating and avoiding detection accompanies the behavior. Simple difference-of-means activation probes detect that signal, catching cases chain-of-thought monitors miss and generalizing to new tasks such as shopping and customer support. On Kimi K3, combining a probe with an LLM monitor cut LLM monitoring cost by 90% with about a 1% drop in precision.

    Why it matters: Goodfire finds a shared internal signal for reward hacking across open-source models and shows activation probes can catch it more cheaply than LLM monitors.

September 7Monday
  1. Charlie O'Neill38

    Let's Bet on RL Environments: Why Useful Data and Environments Will Win

    An argument that people who claim everything will become RL environments are not acting on that belief, and that useful RL environments will grow exponentially scarcer and more valuable as models approach RSI. The piece urges building long-horizon tasks in domains of real expertise rather than Goodhart-style task farms, and argues a broad spread of environments beats a small set of frontier-hard ones.

September 2Wednesday
  1. Charlie O'Neill38

    Trying to Actually Define Continual Learning for AI Agents

    A writer examines continual learning for LLM agents, framing today's models as amnesiac interns whose knowledge resets when the context window ends. LLMs have only two native memories: lossless but costly context and KV cache, and compressed but hard-to-update weights, with little in between. The piece weighs longer context, stateful memory harnesses, and compressing context into learned latent representations, citing Llama 3.1 70B's 141 GB weights and 320 KiB KV cache per token.

August 24Monday
August 20Thursday
  1. Goodfire Research26

    Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation

    Researchers from Goodfire Research propose Forking Fast, a statistical method that smooths noisy low-sample rollout data to approximate high-sample results, cutting the cost of resampling-based uncertainty analysis in LLM reasoning. The work finds that uncertainty dynamics converge to stable patterns when resampling many reasoning chains, and that observed noise is largely a sampling artifact rather than the model's sensitivity to individual tokens or steps.

August 12Wednesday
  1. Modal Blog62

    Qwen3.8-2.4T-A95B 开源权重发布，Modal 同步上线共享端点

    Qwen3.8-2.4T-A95B 已作为开源权重模型发布，并同步在 Modal 上线。Modal 表示其与 Qwen 在发布前合作，通过 SGLang 和针对 Qwen3.8 形状调优的定制 DFlash 推测器，为 Modal Auto Endpoints 提供 day zero 支持。

    Why it matters: Modal 在 Qwen3.8-2.4T-A95B 开源权重发布当天即上线共享端点，并说明其定制 DFlash 推测器如何针对该模型的形状提升推理速度。

July 29Wednesday
  1. Berkeley AI Research58

    From CUDA to MLX: K-Search Brings Kernel Optimization Expertise to Apple Silicon

    IBM Research extended K-Search, the evolutionary kernel search framework from UC Berkeley Sky Lab, with an MLX backend and a structured CUDA-to-MLX translation layer that adapts existing CUDA kernels into Apple Silicon kernels. On an MLX attention kernel the approach reached 0.97x the speed of Apple's native MLX Attention kernel, up from 0.26x without the translation context, and an evolved Mamba SSM kernel reached up to 20x prefill speedup over the community mlx-lm implementation on an M1 Max. The authors attribute the gain mainly to the quality of context and constraints given to the LLM rather than its ability to write Metal code, and say the method is not specific to MLX.

July 26Sunday
  1. Berkeley AI Research33

    ABBEL: Teaching LLMs to Update Belief States for Long-Horizon Interaction

    Berkeley AI Research introduces ABBEL, a framework that replaces full interaction history with natural-language belief states and supervises their contents via belief grading. On CollabBench collaborative coding, reconstruction-based belief grading cuts the gap to full-context models by about 50% and trains in 50 steps instead of 100, while using fewer peak tokens.

July 15Wednesday
  1. Modal Blog60

    Thinking Machines' Inkling multimodal model now available on Modal

    Thinking Machines released Inkling, a general-purpose multimodal model that takes text, image and audio inputs and outputs text, and Modal is offering it from day zero as a Managed Endpoint with token-based pricing. Inkling is a mixture-of-experts transformer with 975B total parameters, 41B active, a 1M token context window and native audio and vision, using five sliding-window attention layers for every full-attention layer. On agentic workloads with 8x B200s, Modal's custom DFlash speculator reached 250 tokens per second per user at 2.5M TPM of per-GPU throughput, 67% faster interactivity than the model's built-in speculative path at matched throughput.

    Why it matters: Modal's day-zero endpoint for Thinking Machines' 975B-parameter Inkling shows how a custom DFlash speculator lifts interactivity 67% over the model's built-in speculative path.

July 6Monday
  1. Modal Blog38

    How to price serverless GPUs: Modal's interactive cost model

    Modal published an interactive tool for estimating serverless versus reserved GPU costs, defaulting to current rates for a B200 GPU serverlessly on Modal and via three-year hyperscaler reservation. The model shows that when an application's peak-to-average ratio exceeds the reservation discount rate, serverless GPUs cost less overall, and Modal cites observed discounts of 2-5x against peak-to-average ratios of 5-10x.

July 1Wednesday
  1. One Useful Thing60

    Ethan Mollick on the twilight of chatbots and the rise of agent-run work

    Ethan Mollick argues that AI use is shifting from chatbot co-intelligence toward long-running agents that need less human intervention, and that this change is happening fast. He cites METR, the UK AI Security Institute and GDPval benchmarks rising faster than exponential, plus an Epoch finding that Opus 4.7 worked alone for 14 hours to build a software package estimated at 2-17 weeks of human engineering for $251 in tokens. An OpenAI and academic economists study found a quarter of OpenAI workers run at least four agents weekly, with legal and HR adopting at nearly the same rate as coders, and a Claude Code study found domain expertise, not profession, predicted success.

  2. Modal Blog58

    Modal Research and HeavyBall's MRP brings multi-token prediction to diffusion LMs

    Modal Research and NYU Shanghai's HeavyBall Research introduce Multi-token Residual Prediction (MRP), a 3-layer module attached to a frozen diffusion LM backbone that predicts the inter-step logit residual instead of the full next-step distribution. On SDAR-4B GSM8K, MRP holds 70.9% accuracy at K=3 versus 5.9% for naive MTP distillation, and in SGLang it reaches up to 1.56x throughput losslessly in speculative mode or recovers up to +16 accuracy points under aggressive low-threshold dynamic decoding. The paper, code, SGLang implementation, and SDAR-MRP models on Hugging Face are released.

June 25Thursday
  1. Modal Blog55

    Modal launches serverless Servers for low-latency HTTP, WebSocket and gRPC traffic

    Modal launched Servers, a regionalized autoscaling pool of HTTP server replicas for HTTP, WebSocket and gRPC traffic aimed at latency-sensitive workloads such as LLM inference for interactive agents. Modal says it cut p50 latency from 39ms to 6ms by keeping control-plane lookups and queueing off the request path, so clients get a 503 when no replica is available and must handle load shedding in the application. The routing layer runs on AWS EKS with Envoy at the edge and Modal's in-house Rust proxy fprs, built on Cloudflare's pingora, handling domain association and load balancing, with config read from Spanner and cached in memory.

June 24Wednesday
  1. Modal Blog58

    Modal launches Auto Endpoints, citing speculative decoding for state-of-the-art inference latency

    Modal is launching Modal Auto Endpoints, which bring Modal's infrastructure scalability to state-of-the-art inference performance with a single click while keeping control of the code. Modal says the main latency win comes from speculative decoding with custom DFlash speculator models, which cut about 100ms of end-to-end latency in a Decagon Voice case study, roughly 40% of server-side decode latency, making it over 60ms faster than the fastest proprietary alternative. Modal also released open source DFlash speculator models with Z Lab and SGLang, including one for Qwen 3.5 397B-A17B that improves on MTP by over 50%.

June 23Tuesday
  1. Modal Blog60

    Modal launches Auto Endpoints for self-serve, OpenAI API-compatible LLM inference

    Modal launched Auto Endpoints, a self-serve service that deploys production-grade LLM inference as an OpenAI API-compatible endpoint backed by a Modal App the user can see and control. Each supported model ships with a starting deployment tuned from Modal's own benchmarking, using engines like SGLang and the DFlash speculative decoding drafter from Z Lab, and users can inspect GPU choices, engine flags and inference metrics such as TTFT, ITL and speculative decoding acceptance length. Modal also released Modal Servers from beta for regionalized routing with 5ms overhead, and said it generates endpoint deployments with an internal agentic system still monitored by human engineers.

    Why it matters: Modal's Auto Endpoints expose the serving stack, engine metrics and benchmark recipes that managed inference providers keep hidden, giving teams a self-serve path to owning open-model inference.

June 19Friday
  1. Modal Blog60

    Modal releases DFlash speculators for Qwen 3.5 and 3.6, claiming 5-20% extra speedup

    Modal released a set of DFlash draft models for the Qwen series on Hugging Face, including Qwen 3.6 35B-A3B-DFlash and Qwen 3.5 variants from 4B to 122B-A10B, plus a DFlash speculator for Qwen 3.5 397B-A17B built with SGLang. The company says the new drafters add 5-20% speedup over existing DFlash baselines and can run Qwen 3.5 122B-A10B at over 1000 tok/s at concurrency 1 on a B200 node, versus 250 tok/s without speculation. The post argues speculative decoding delivers 2x-3x gains where kernel optimization yields small percentage points, and presents a roofline model for estimating speedup from acceptance length.

    Why it matters: Modal explains why it bets on speculative decoding and releases DFlash draft models for the Qwen series, with measured speedups and a roofline model for estimating gains.

June 11Thursday
  1. Modal Blog34

    Modal Makes FlashAttention-4 Faster for LLM Inference

    Modal contributed optimizations to the FlashAttention-4 kernel to speed up decode-heavy LLM inference workloads, where memory bandwidth is the bottleneck. The changes include porting the "split KV" parallelism technique to FA4 and replacing TMA cp.async.bulk loads with cp.async for irregular memory accesses. An FP8 input PR (merged April 17, 2026) delivers up to 1.16x throughput over the bf16 baseline and enables smaller KV caches.

June 10Wednesday
  1. One Useful Thing78

    Ethan Mollick reviews Claude 5 Fable, the first public Mythos-class model

    Ethan Mollick tested Claude 5 Fable, the first Mythos-class model released to the public, and found it outperformed every other public model he has used by a considerable margin, running up to a dozen hours on multi-page specifications. In one Claude Code session it built an interactive isochrone map by spawning cheaper Claude Sonnet agents to research over 2,200 flights, rail schedules and road speeds, and in another it worked nine and a half hours to produce Concord, software for calibrating human and AI judgement.

    Why it matters: An early hands-on account of a Mythos-class model that shows how multi-hour autonomous agent runs shift the human role from steering to commissioning outcomes.

June 9Tuesday
  1. MiniMax Blog58

    MiniMax details MaxProof: generative-verifier RL and evolutionary search for mathematical proof

    MiniMax published a technical blog explaining MaxProof, the framework behind M3 exceeding the human gold-medal threshold on IMO 2025 and USAMO 2026. The pipeline trains three expert models for proof generation, error judgment and proof repair, then runs an evolutionary test-time search that samples 32 candidates, verifies each 4 times and refines over up to 10 rounds. MiniMax says the key lesson is that a generative verifier used as an RL reward should prioritize low false positives over average accuracy, citing M2-cycle reward hacking where the false-positive rate rose from about 2.9% to 8.0% during training.

May 12Tuesday
  1. Modal Blog58

    How Modal achieved truly serverless GPUs

    Modal published an engineering account of how it cut AI inference replica spin-up from roughly 2,000 seconds to about 50 seconds using four components: a buffer of idle healthy GPUs, a lazy content-addressed filesystem called ImageFS, host-side checkpoint/restore via gVisor's runsc, and CUDA checkpoint/restore built on Nvidia driver support. Across over ten thousand cold starts, mean vLLM boot latency fell from 95,679 ms to 13,797 ms and SGLang from 83,713 ms to 17,486 ms for a ~1 GiB Qwen 3 0.6B model in bf16. The post also notes limits, including single-GPU-only snapshots and host-environment sensitivity that requires multiple snapshots per deployment.

May 8Friday
  1. Berkeley AI Research38

    Adaptive Parallel Reasoning: A Survey of Efficient Inference Scaling

    A Berkeley AI Research survey examines adaptive parallel reasoning, in which a reasoning model itself decides when to decompose subtasks, how many concurrent threads to spawn, and how to coordinate them. The post contrasts this with fixed parallelism methods such as self-consistency, Best-of-N, Tree of Thoughts, MCTS, ParaThinker, GroupThink and Hogwild! Inference, which impose the parallel structure externally. Sequential reasoning scales linearly in tokens and latency and risks context-rot, motivating parallel threads that explore independently and concurrently.