Skip to content
#Data / training
November 3Tuesday
  1. Apple Machine Learning Research32

    Attention to Mamba: A Recipe for Cross-Architecture Distillation

    Apple researchers propose a two-stage recipe for distilling a Transformer into a pure Mamba architecture without any Attention blocks. The method first distills a pretrained Transformer into a linearized Attention model via an adapted kernel trick, then distills that into an adapted Mamba model. The resulting distilled Mamba retains most of Pythia-1B's performance, reaching 14.01 perplexity versus the teacher's 13.86, with some gaps on tasks relying on in-context recall.

November 2Monday
  1. SemiAnalysis58

    SemiAnalysis releases GDPx benchmark for frontier agentic capabilities across high-value domains

    SemiAnalysis released GDPx, a benchmark covering chip design, cybersecurity, biology, finance, legal, quant trading, RSI, hardware design, materials science, alignment and software engineering, built with 14 specialized RL environment partners. Even the best model scores just 12%, with Opus 5.5 winning quant trading, hardware and chip design, Gemini 4 Carbon leading biology and software engineering, and GPT-6 Astra topping RSI and materials science. Every task runs in a realistic simulated environment, such as a ~4,000-file private equity deal room or live multi-service cyber ranges, unlike OpenAI's GDPval, which SemiAnalysis argues spoon-feeds context and lacks environments.

November 1Sunday
  1. Sakana AI Blog40

    Sakana AI Symposium: Jürgen Schmidhuber on AI's Past and Future, Japan's Push into Language Models, World Models and Physical AI

    Sakana AI held its "From World Models to Real-World AI" symposium in Tokyo on October 26, 2026, where Chief Scientific Advisor Jürgen Schmidhuber gave a keynote and Sakana AI announced it will develop its own pretrained language model series, Sakana Sawara (tentative name). The first small model, Sawara Small (tentative name), is planned for release in spring 2027 if development proceeds smoothly. More than 400 attendees joined the event, which also featured a conversation between Schmidhuber and CEO David Ha.

  2. Recode China AI22

    CXMT Pushes Into HBM3E, Unitree Q3 Revenue Up 70%, and China's Stock Slide

    CXMT reported Q3 net profit of RMB54.4 billion and revenue of RMB118.71 billion, up 613.2% year on year, while Reuters says it has sent HBM3E samples to domestic chip designers with limited production planned within two months despite yields of just 20–30%. Unitree's first post-listing results showed Q3 revenue up 69.9% to RMB665 million as research spending rose roughly 250%. Moonshot launched Kimi K3.1 Pro with a one-million-token context window, then apologized for rate limits and restored a quarter of subscribers' monthly allowance.

October 30Friday
  1. Microsoft Research40

    Microsoft builds synthetic enterprise data platform to evaluate Copilot

    Microsoft's Industry Evaluation Data Generation Platform creates full-scale synthetic enterprise environments to evaluate Microsoft Copilot without using customer data. Grounded in public sources such as O*NET occupational data, job postings, and SEC filings, it generates realistic tenants with users, meetings, emails, and documents, plus automated query pipelines and quality frameworks. Microsoft says this establishes the first standardized evaluation baseline for Copilot.

  2. ChinaTalk38

    Is Half of China's AI Compute Really Being Spent on Slop?

    A viral claim that 55% of China's AI tokens go to short dramas and video generation — traced to a June investor call by COL Group — has drawn rebuttals from ByteDance, which says video generation is only about 10% of its own compute, and from People's Daily. Over 90% of the roughly 430,000 micro-dramas released in the first eight months of 2026 were AI productions, and a five-second 720p clip on ByteDance's Seedance costs around 100,000 tokens.

  3. Microsoft Research22

    Microsoft Research Focus Edition #97: ForkPilot, CABRA, TeleTune and AI Use at Work

    Microsoft Research Focus Edition #97 highlights ForkPilot, a self-improving agentic framework that matches the best existing methods while cutting AI processing demands by up to 59% across six benchmarks. The edition also covers CABRA, a framework showing coding agents still struggle to understand unfamiliar codebases, and TeleTune, which learns reusable skills from user activity logs. It also examines Copilot Chat activity, where document editing exceeded 20% of classified activities.

  4. Amazon Science38

    Speeding up LLMs for travel applications: Medusa plus knowledge distillation

    Combining knowledge distillation with the Medusa framework lets a 1.1B-parameter TinyLlama student match the quality of its 70B-parameter Llama-3.1 teacher while running 10 to 20 times as fast at less than one-third the cost. Working with Booking.com on AI Trip Planner, Smart Filters and accommodation description generation, the pipeline closed over 90% of the quality gap and added 2 to 3.6x further speedup from Medusa heads.

October 29Thursday
  1. Meta Engineering Blog40

    Meta Andromeda: Meta's Next-Gen Ads Retrieval Engine Hits 1B Ads per Second

    Meta detailed the next generation of Andromeda, its personalized ads retrieval engine, which scores over one billion ads per second in production — a 100x throughput lift over the 2024 system. The new architecture brings LLM-inspired mixture-of-experts to retrieval, approaching ranking-level model performance at roughly 300x fewer inference-time FLOPs, with over 10 billion ads per second on a single NVIDIA GH200 GPU. Since 2025, Andromeda has delivered a 2.4% increase in conversions on Facebook.

  2. Preferred Networks News40

    Zenrin Group and Preferred Networks Form Business and Capital Alliance to Fuse Geospatial Data with AI

    Zenrin, Zenrin DataCom and Preferred Networks have signed a business and capital alliance, with the Zenrin companies subscribing to a PFN third-party allotment. The partners will combine continuously updated geospatial data, including historical change records, with PFN's generative AI, physical AI and 3D spatial recognition technology to build AI models that understand and predict real-world change. Initial focus areas include public sector, real estate, retail and distribution, targeting demand forecasting, store site selection and urban planning, disaster prevention and logistics simulation.

October 28Wednesday
  1. Ai2 · Allen Institute for AI40

    How researchers are adapting olmOCR 2 for historical and archaeological archives

    Researchers are adapting Ai2's open olmOCR 2 model to transcribe historical and archaeological archives. Historian Maurice Casey built ArtificeOCR, which runs olmOCR 2 locally with Tropy to make his archival photo library keyword-searchable, while TU Wien's CENTURIA project fine-tuned olmOCR 2 with LoRA on 57 annotated records, cutting transcription error on 450 test records from 15.58% to 1.11%.

  2. Google Research58

    Google Research's sleep framework lets LLMs consolidate new knowledge without catastrophic forgetting

    Google Research introduced a sleep framework for continual learning that alternates active interaction with offline memory consolidation, letting LLMs absorb new facts into expanded Mixture-of-Experts parameters instead of losing prior knowledge. The approach combines parameter expansion, on-policy distillation with reinforcement learning-based imitation, and a dreaming stage that generates and filters synthetic training samples. The authors report near-zero performance drop in continual translation for unseen languages such as Manchu and Kalamang, near-perfect scaling to 10M tokens on BABILong, 48.9% single-task and 46.2% CPT results on SQuAD-based factual incorporation, and 79.2%, 69.0% and 46.1% on AIME 24/25 and HMMT 25.

  3. Meta AI Research Blog30

    Meta Supports AI Research at Five US Universities to Accelerate Medical and Scientific Breakthroughs

    Meta is supporting research at five US universities — Caltech, Ohio State University College of Medicine, Louisiana State University, Indiana University, and UTHealth Houston — with access to its latest Muse models, researcher collaboration, and funding. Researchers will use Muse Spark Open, Muse Wonder, Muse Spark, the 30B open-weight Muse Glimmer, plus SAM, V-JEPA, DINO, UMA, and SeamlessM4T. Projects span medicine, physics, chemistry, biology, computer science, and mathematics, with universities retaining control of all patient and clinical data.

October 27Tuesday
  1. Ai2 · Allen Institute for AI58

    Ai2 study tests whether LLMs can reliably judge research idea novelty

    Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.

  2. Redwood Research22

    Redwood Research argues for accelerating AI's conceptual reasoning capabilities

    Redwood Research argues that AI's conceptual reasoning capabilities—tasks like philosophy that lack verifiable ground truth—should be accelerated differentially, since AI R&D and takeover rely far less on such reasoning. The team says basic elicitation techniques and small-scale training can make models three to twelve months better on its main conceptual reasoning benchmark, and that the area is currently neglected and underinvested by the AI safety community.

  3. Epoch AI · Gradient Updates58

    Epoch AI tests AI on its own research tasks, top models score 65% and fail all

    Epoch AI expanded its earlier effort to automate a senior researcher's job into the Epoch Automation Reports, evaluating AI on real Epoch tasks like graphic design and research planning against the same standards used for human employees. 1 and GPT-6 Astra averaged 65% on the rubric, but a 65% score still counts as a failed task, since every submission would be rejected or revised if a human had made it.

  4. Anthropic Research65

    Anthropic reports early progress on using Claude to accelerate drug development

    Anthropic's life sciences team has spent several months working with six drug developers, including Bristol Myers Squibb, Generate Biomedicines and Divergence Labs, plus running its own discovery programs, to test where Claude can speed up drug development. Reported early results include a Generate design roughly 70% shorter than the standard version that let immune cells make about 10 times as many receptors, and roughly 35,000 antibody designs generated largely autonomously for Divergence in about a month. Anthropic says the work is still mostly preclinical and plans to extend Claude into clinical trials as the programs advance.

    Why it matters: Anthropic's first detailed account of embedding Claude in real drug programs shows where the model helps and where it still falls short across six partners.

October 24Saturday
  1. SemiAnalysis60

    SemiAnalysis tests Amazon Trainium3's native PyTorch training stack and Neuron Explorer profiler

    SemiAnalysis's engineering team tested Trainium3 racks on the TorchTitan and TorchNeuron stack and found the native PyTorch experience a major improvement over the old PyTorch/XLA approach, though bugs and long compile times remain. Training Qwen3-32B on 16 Trainium2 chips reached 14,156 tokens per second at 30.47% MFU in compiled mode, a 1.28x gain over eager mode, while the gpt-oss 20B recipe took 27 minutes and 17 seconds to compile. The team also rated Neuron Explorer as the industry's leading profiler, ahead of Nvidia's and AMD's tools, and reported Trainium3 passed 89 of 96 FlexAttention test cases.

October 23Friday
  1. Ai2 · Allen Institute for AI31

    LitPivot: Ai2's System Helps Researchers Pivot Research Ideas Through Literature

    Ai2 and University of Pennsylvania researchers built LitPivot, a system that organizes retrieved papers around the facet of a research proposal being edited and suggests literature-backed revisions. In a study with 17 researchers, domain experts rated ideas developed with LitPivot as better grounded in the literature than those built with a text editor and an AI chat tool. Participants also reported their revised ideas as more novel and useful, and LitPivot will be presented at UIST 2026.

  2. Amazon Science38

    The science behind TorchNeuron, Amazon's new PyTorch backend for AWS Trainium chips

    Amazon released TorchNeuron, an open-source PyTorch backend that translates PyTorch code into operations executable on AWS Trainium systems with minimal or no code changes. It adds an opt-in adaptive eager mode that groups and compiles operations for near-compiled performance, plus multi-level NEFF caching; a shared compilation service cut time to first training step by about a third in a 64-process LLM test. The Annapurna team tuned and benchmarked it using 841 representative models distilled from hundreds of thousands of open-source models.

October 22Thursday
  1. Modal Blog71

    Modal releases Spindle, an open-source Tinker-compatible training server

    Modal released Spindle, an open-source, Tinker-compatible training server that runs in a customer's Modal workspace, letting teams customize the stack and pay for GPU time instead of tokens. Spindle supports full-parameter training and multi-LoRA across open models, and Modal reports that sharing one trainer across 8 clients cut cost per 1M output tokens by up to 80% versus Tinker on a DAPO-math GRPO job. Modal says Spindle is still in beta and positions it between its Dojo training library and raw cluster primitives.

    Why it matters: Modal's open-source Spindle server offers teams a Tinker-compatible alternative that trades per-token pricing for GPU time, with multi-LoRA sharing reported to cut costs by up to 80%.

  2. Ai2 · Allen Institute for AI58

    Ai2 releases OlmoEarth v1.3 satellite embedding model with global precomputed embeddings and a browser explorer

    Ai2 released OlmoEarth v1.3, a satellite embedding model that turns a year of Sentinel-1, Sentinel-2 and Landsat time series into one 128-dimensional vector per 10-meter pixel, with the first 64 dimensions usable as a smaller embedding. Ai2 precomputed annual embeddings for the whole planet and published them on Source Cooperative, and its 128-dimensional embeddings scored highest on 20 of 27 evaluations against AlphaEarth Foundations and Tessera v2, including a new tropical crop benchmark called PLANTEUR. The release also includes the OlmoEarth Embeddings Explorer, a no-sign-in browser tool for comparing years, finding similar places and training small classifiers on the spot, with 2025, 2024 and 2023 global and 2017 to 2022 covering select countries; weights, pretraining data and code are open.

  3. inclusionAI Models58

    inclusionAI releases Ling-3.0-flash-Sante, a 124B healthcare-specialized MoE model

    inclusionAI released Ling-3.0-flash-Sante, a healthcare-specialized model built on Ling-3.0-flash with 124B total parameters and 5.1B active parameters per token, with weights on Hugging Face and ModelScope. It uses a hybrid-linear MoE architecture combining Kimi Delta Attention, gated MLA and sparse MoE layers, trains to 256K tokens, and enables thinking mode by default with tool calling through compatible serving frameworks. The company labels it a research model, not a medical device, and says outputs must not replace professional medical advice, diagnosis or treatment.

  4. Microsoft Research64

    Microsoft Research and co-author prove the 31-year-old KLS conjecture

    Microsoft researcher Xinzhi Zhang and a co-author have proved the Kannan-Lovász-Simonovits (KLS) conjecture, a 31-year-old open problem in high-dimensional geometry, showing the bound holds with a universal constant independent of dimension. The proof grew out of a human-directed agentic process in which AI agents explored more than 100 candidate approaches while researchers pruned and redirected the search, and a separate unguided attempt with leading models on a comparable token budget fell short. Two other proofs appeared the same day, and a Lean 4 formalization was subsequently released.

    Why it matters: A 31-year-old open problem in high-dimensional geometry is resolved through a human-directed agentic search over roughly 100 candidate approaches, with independent proofs and Lean formalization as confirmation.

  5. Anthropic Newsroom28

    Anthropic Opens Call for Guest Investigators in the Life Sciences

    Anthropic is inviting principal investigators and their research groups at academic institutions, nonprofits, and government labs to apply for 8–12 week guest investigator projects using Claude in areas such as causal human biology, structural and computational molecular biology, medicinal chemistry, drug discovery, and clinical science. Anthropic will run its models and harnesses at no cost, and outputs will be published with open-source data and code while investigators retain rights to their data. Applications are open for approximately 14 days, with selected participants notified by email about 14 days after the close.

October 21Wednesday
  1. Modal Blog58

    Modal releases Dojo, a library for production-grade LLM training

    Modal released Modal Dojo, a training library that provides building blocks, tuned default recipes and an observability dashboard for production-grade LLM post-training. It builds on open-source frameworks Miles and Slime, supports models from half a billion to multi-trillion parameters, and exposes an API and CLI that agents can drive to launch and debug runs. Modal says Axiom Bio used it to scale to 64k-context DeepSeek V4.1 Flash runs on 64 B300s, and Afterquery trains Kimi and DeepSeek at 256k context length.

  2. Apple Machine Learning Research26

    TEAL: Token-Space Efficient Active Learning for Low-Budget Semantic Segmentation

    Apple researchers (with University of Waterloo) introduce TEAL, the first pixel-level active learning framework built on frozen ViT representations, selecting diverse candidates directly on the native token lattice via a two-level MaxHerding strategy plus margin-based uncertainty refinement. With a single labeled pixel per image, TEAL beats the strongest baseline by up to +24.36 mIoU on CamVid, Cityscapes, ADE20K and Pascal Context, reaching up to 97% of fully supervised performance with just 10 px.

October 20Tuesday
  1. Google Developers Blog60

    Google 将 TorchTitan 接入 TorchTPU，并发布 TPU 模型优化手册

    Google released a TPU fork of Meta's TorchTitan that runs PyTorch training on Cloud TPUs through TorchTPU's native torch.device("tpu") backend, with TPU-native kernels and a step-by-step optimization playbook. On TPU v7x (Ironwood), Llama 3 405B holds about 57% MFU from 128 to 1,024 chips, Qwen3 235B reaches 32.42% MFU on one 1,024-chip slice, and FLUX.1 reaches 19.4% MFU on 1,024 chips at 93% weak-scaling efficiency. The fork is at https://github.com/google-pytorch/torchtitan/, and a Transformers modeling backend plus an agent onboarding skill extend the same trainer to other checkpoints.

    Why it matters: Google 与 Meta 合作把 TorchTitan 移植到 TPU，并给出从 FLUX.1 到 Qwen3 235B 的逐级调优路径与 MFU 数字，可作为 PyTorch 团队评估 TPU 训练可行性的参考。

October 17Saturday
  1. Google Research71

    Google Research outlines VERID and closed-loop agent workflows for autonomous wet labs

    Google Research describes three projects aimed at grounding AI agents in real wet-lab experimentation, including the VERID framework, a semantic digital twin of a lab workcell built on the open-source PyLabRobot framework that tracks deck layout, well volumes and tip inventory and dry-runs protocols before execution.

    Why it matters: Google Research outlines three projects that connect AI agents to real wet-lab hardware, showing how simulated dry runs and closed-loop experiments let agents correct their own design rules.

  2. Big Technology42

    Rivian CEO RJ Scaringe: Robotics Is Progressing Much Faster Than You Think

    Rivian CEO RJ Scaringe says robotics progress will be "far, far, far, far faster" than the world expects, citing robots that now learn broad, generalizable skills through transformer-based models. His 2025-founded startup Mind Robotics raised $1 billion to build two-armed industrial robots sold to manufacturers, and it trains the technology inside Rivian's plants.

October 15Thursday
  1. Mistral AI71

    Mistral releases Timestral time-series foundation models, topping GIFT-Eval, TIME and fev-bench

    Mistral released Timestral, its first family of time-series foundation models, in four sizes from 5M to 209M parameters. The largest model ranks first on all three main public benchmarks, GIFT-Eval, TIME and fev-bench, among pre-trained zero-shot models from a single checkpoint without ensembling, using two-thirds of the parameters of the previous state of the art. Timestral produces probabilistic multivariate forecasts with 21 quantiles, supports past-only and known-future covariates, runs on CPU without a GPU, and is available through Mistral Forge for fine-tuning, with a GitHub repository and technical report published under a modified MIT licence.

    Why it matters: Mistral's first time-series foundation model family claims top zero-shot rankings on GIFT-Eval, TIME and fev-bench with a 209M-parameter checkpoint, offering a size-accuracy tradeoff reference for forecasting work.

  2. Goodfire Research62

    Goodfire research essay argues neural networks are already clean and superposition was misread

    Goodfire researcher argues that the canonical evidence for polysemanticity and superposition in neural networks was misread, and that what looked like arbitrary packing is mostly a motif he calls recursive generalization. Retraining InceptionV1 with ordinary architectural improvements and no change to model size or training data made 99-100% of neurons in its last layer easily interpretable, and preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job. The essay concludes networks use one learning principle rather than two, so interpretability can proceed by studying neurons and circuits directly instead of first training decomposition models such as sparse autoencoders.

    Why it matters: A first-hand reinterpretation of the field's canonical polysemanticity evidence, arguing that networks generalize recursively rather than pack unrelated concepts, which would make direct mechanistic study viable.

October 13Tuesday
  1. Modal Blog58

    Modal tests mixed-precision, mixed-accelerator RL samplers on Qwen3.6-35B-A3B

    Modal trained Qwen3.6-35B-A3B on agentic coding with samplers spread across six accelerator types (B200, B300, H100, H200, A100, RTX PRO 6000) at NVFP4, FP8 and BF16 precision, feeding one BF16 trainer on B200. Plain GRPO collapsed on both mixed and homogeneous samplers, but adding masked importance sampling plus score centering, with or without top-p mask replay, trained stably to the end and reached SWE-bench Pro V2 pass@4 on par with homogeneous samplers, at roughly 15% lower sampler cost per update. The mixed samplers ran with an average sampler-trainer KL 4.7x that of the homogeneous setup, and the authors argue stability depends on keeping the mismatch bounded rather than low.

  2. Amazon Science75

    Amazon Web Services researchers propose priming to convert pretrained Transformers into hybrid state space models without full retraining

    Amazon Web Services researchers introduce priming, a method that reuses pretrained Transformer weights to cut the pretraining cost of hybrid state space architectures by more than 99%, using under 0.5% of the source model's pretraining token budget. In controlled comparisons, Gated KalmaNet outperformed Gated DeltaNet and Mamba-2, and a 32B GKA-based hybrid matched its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context. The team released 8B and 32B primed hybrid models, training code, Triton kernels, and a vLLM serving plugin under the Apache 2.0 License.

    Why it matters: Amazon Web Services researchers show that reusing pretrained Transformer weights can cut hybrid architecture pretraining cost by over 99%, making controlled comparisons of state space layers practical.

October 10Saturday
  1. Epoch AI · Gradient Updates65

    Epoch AI's InnovationEval finds frontier agents reach only 15% of SDPO's gains on automated AI R&D

    6 Sol and Fable 5 with independently rediscovering a novel post-training technique comparable to Self-Distillation Policy Optimization (SDPO), giving each 3,000 GPU-hours in a sandbox without internet access. 6 Sol's method, largely copied from prior work, reached 15% of SDPO's gains after adjusting for slower training, while Fable 5's technique failed to improve performance.

October 9Friday
  1. Epoch AI · Gradient Updates22

    Epoch Brief, October 8, 2026: AI Agent Population Estimates, Falling AI Costs, and China Chip Exposure

    Epoch AI estimates that AI chips shipped through 2027 could support about 30 to 170 million concurrent frontier-model agents, or almost 2 billion if run on cheaper models. Its research also finds the cost of a fixed level of AI performance has fallen about 47% per quarter over three years, roughly 13× per year. Separately, Epoch estimates China's exposure to semiconductor supply shocks is 2.7× that of the US.

October 8Thursday
  1. Zyphra Research36

    Zyphra Research: Expert Coupling Cuts MoE Pretraining Communication

    Zyphra Research shows MoE routing is highly structured, with co-selected experts correlated within a layer and a token's expert choices predicting later-layer routing. Exploiting this via correlated expert placement and token shuffling cuts all-to-all communication time up to 2.63× and improves end-to-end training step time up to 1.41× in Megatron-LM experiments on MI300X GPUs, without changing architecture or routing decisions.

  2. Waymo Blog30

    Waymo Study: Sober Drivers Still Face Nearly 4x Nighttime Fatal Crash Risk

    Waymo's peer-reviewed research finds that even fully sober drivers face 3.1 to 3.9 times higher nighttime fatal crash risk than in daytime, and that removing alcohol-impaired drivers cuts average urban fatal crash involvement by 23%, from 1.42 to 1.10 per 100 million miles. A second study of 37,654 fatal cases in the 2023 FARS database shows only 8.2% involved strictly human behavior factors alone. Waymo frames the findings as support for a Safe System Approach and for benchmarking autonomous driving against a zero-BAC human baseline.

October 3Saturday
  1. Epoch AI · Gradient Updates58

    Epoch AI: Compute built over the next two years could run hundreds of millions of AI agents

    Epoch AI estimates that AI compute produced over the next two years could support tens to hundreds of millions of concurrent agents running today's most capable models, or billions running cheaper ones. Using projected HBM shipments from 2025 through 2027, the analysis counts GB300-equivalent GPUs and finds top-tier agents running nonstop could supply the weekly hours of roughly 140 million to 700 million full-time employees, while DeepSeek V4 Pro serving benchmarks imply about 1.9 billion concurrent agents. The report notes the key uncertainty is whether demand grows fast enough to justify the buildout or leaves a glut of agents.