Skip to content
#Multimodal
November 9Monday
  1. inclusionAI Repositories38

    ZoomGain: Mining Global-to-Zoom Perception Gaps for Fine-Grained Visual Supervision

    ZoomGain constructs questions about a full image from visual details that only become clear when zooming in, using this global-to-zoom perception gap as supervision for vision-language models. The release includes the arXiv paper, the ZoomGainBench benchmark (1,000 questions over 941 images covering attribute recognition, counting, and OCR), 9,000 RL training records over 7,713 source images, and the ZoomGain-4B and ZoomGain-9B checkpoints. The best of 28 models scored 61.08 on ZoomGainBench, while ZoomGain-9B reached superior performance on multiple fine-grained and general perception benchmarks.

  2. Meta AI Research Blog78

    Meta releases Muse Ember, its smallest and most efficient model for agent tasks

    Meta released Muse Ember, its smallest and most efficient model, aimed at the repetitive work agents do all day such as tool calls, coding subtasks, routing, extraction and long document parsing. Meta says it matches Claude Haiku 5.5's best score at about a third of the cost and beats every GPT-6 Luna setting, at $0.10 per million input tokens and $0.50 per million output tokens. It ships with a 1M-token context window, up to 512K output tokens, native text, image, audio and video input, and is available today on Muse Code and the Meta Model API as muse-ember-1.0.

    Why it matters: Meta's smallest model targets the high-volume agent steps, and its cost-per-task comparison against Claude Haiku 5.5 and GPT-6 Luna shows where the efficiency claim comes from.

November 8Sunday
  1. Recode China AI22

    Apple Tests Qwen in Apple Intelligence in China as Tencent Adds Vision to Hy4

    Apple began developer testing of Qwen inside Apple Intelligence in China this week, ahead of a planned December rollout with iOS 27.3, bringing Qwen into Siri, Writing Tools and Visual Intelligence on an opt-in basis. Tencent's November 4 Hy4 release adds visual reasoning and computer use, while Baidu, Moonshot, Ant and Meituan also published or launched new models.

November 6Friday
  1. vLLM Blog55

    VeRL-Omni v0.3.0 released with async training and broader multimodal post-training

    VeRL-Omni v0.3.0 was released with asynchronous and decoupled RL execution paths, the V1 trainer as the default, and expanded post-training coverage across image, video, audio, and multimodal understanding. The release adds fully async Qwen3-Omni text and audio training, separate-async Omni and diffusion training with a replay buffer, and new recipes including MiniMax H3 with DiffusionNFT and MiniCPM-o 4.5 GSPO. In a MiniCPM-o 4.5 AVQA experiment on 4×H800 80 GB, separate-async 3+1 was about 9% slower per prompt than colocated 4+0 at natural response length, roughly tied at 512 tokens, and about 6.7% faster at 1024 tokens.

  2. Meta Newsroom · AI62

    Meta rolls out real-time voice and expressive avatars in its Muse app

    Meta is rolling out real-time voice conversations and expressive avatars in the Muse app on mobile, web and Mac, starting in the US. Users tap the phone icon to talk, can interrupt responses, and choose from hundreds of voices plus voice-only, avatar or full-screen modes. Voice is coming soon to AI Glasses and Muse Charm.

    Why it matters: Meta's Muse app now supports real-time voice conversations with expressive avatars, showing how voice and interruption handling are moving into consumer AI assistants.

  3. Google DeepMind78

    Google DeepMind releases Gemini Omni 1.2 Flash with 15-second clips and 30% lower price

    Google DeepMind rolled out Gemini Omni 1.2 Flash, now generally available across the Gemini API and Gemini for business, with native 15-second video generation, multi-turn conversational editing and stronger reference following. The price drops 30% from Gemini Omni 1.1 to $0.07 per second for 720p video, and users can draft at 360p or 720p before upscaling to 1080p or 4K. Clips can be set from 3 to 15 seconds in one-second increments, and outputs carry an imperceptible SynthID watermark.

    Why it matters: Gemini Omni 1.2 Flash raises native clip length to 15 seconds and cuts the price 30% to $0.07 per second of 720p video, with multi-turn conversational editing now generally available.

  4. Google Blog · AI80

    Google releases Gemini Omni 1.2 Flash with 15-second clips and a 30% price cut

    Google DeepMind rolled out Gemini Omni 1.2 Flash and made it generally available across the Gemini API and Gemini for business. The model generates and edits native clips up to 15 seconds, up from 10 seconds in Omni 1.1, supports multi-turn conversational editing, and holds character, product, and style references more consistently; 720p video now costs $0.07 per second, 30% less than Omni 1.1. Outputs carry a SynthID watermark, and the model is rolling out in Flow, Google Vids, the Gemini app, and Google Ads.

    Why it matters: Gemini Omni 1.2 Flash cuts the price 30% to $0.07 per second for 720p video while extending native clips to 15 seconds, giving teams producing video at scale a cheaper option to compare against prior versions.

  5. Artificial Analysis Articles40

    VIALS-AA: Different Paths to Scientific Image Reasoning

    VIALS-AA analyzes how 19 models reason over scientific images, classifying 9,177 rollouts of tool calls. Viewing images accounted for 38.3% of calls and image preparation or measurement 47.3%, while image measurement was just 5–6% of calls for GPT-6 Astra (max) and GPT-6.1 Sol (max) versus 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). Gemini 3.8 Flash (high) produced 86.7% of the cohort's commentary/no-op calls, and cost per task varies by nearly 100× across models.

  6. Descript Changelog22

    Descript November 6, 2026 Release Roundup: Brand Layout Packs, Underlord Skills, 30-Day Trash

    Descript's November 6, 2026 roundup adds generative layout packs that turn brand guidelines, logos and colors into four AI-proposed directions and a full editable layout pack on paid plans. Underlord now supports reusable skills, invoked with "/" in chat, replacing prompt templates, which switch off on November 30. Deleted projects now sit in trash for 30 days, and Enterprise drive search matches what's inside images and videos, including via the Descript MCP and API.

November 5Thursday
  1. Google Research47

    Google Research's Agile Deliberation helps humans define subjective AI categories

    Google Research introduced Agile Deliberation, a framework that turns subjective AI classifier creation into an iterative conversation where the AI helps users sharpen their own definitions. It splits the task into concept scoping and concept iteration, using vision-language models to surface borderline images and route feedback into an automatic prompt optimization loop. In 18 live 1.5-hour user sessions, the approach reduced user fatigue and anxiety and produced more consistent models that outperform standard baselines; the code is open-sourced on GitHub.

  2. Preferred Networks News71

    PFN releases PLaMo 3 Translate open-weight translation LLM for edge devices

    Preferred Networks released PLaMo 3 Translate, a translation-specialized LLM used in its PLaMo Translation service, as an open-weight model on Hugging Face on November 5. 8B, 2B, 8B and 31B) and supports 53 languages including Japanese and English, running on smartphones, laptops and internal servers without a cloud connection.

    Why it matters: PFN releases a translation-specialized LLM in four sizes from 0.8B to 31B, letting companies and individuals run 53-language translation on-device without cloud access.

October 30Friday
  1. Meta Newsroom · AI25

    Meta Partners with INCA to Expand AI Glasses Access for Blind and Low-Vision Canadians

    Meta has partnered with INCA, a Canadian nonprofit founded in 1918, to help more blind, deafblind and low-vision people in Canada benefit from AI glasses. Meta funding will support INCA in four areas: listening to the community, backing community champions who run in-person demos and peer training, feeding community feedback into Meta's accessibility and product teams, and providing accessible learning guides and training resources. Meta is also donating AI glasses through INCA's "Opening the Doors to Work" program for people seeking employment.

  2. Meta Newsroom · AI25

    Meta and CNIB Partner to Bring AI Glasses to Blind and Low Vision Canadians

    Meta has partnered with CNIB, a non-profit founded in 1918, to bring AI glasses to more Canadians who are blind, Deafblind or have low vision. Meta funding will support CNIB in four areas: community listening, peer training led by community champions, sharing insights with Meta's accessibility and product teams, and accessible guides and training materials. Meta is also donating AI glasses through CNIB's Come to Work program for people exploring employment.

October 29Thursday
  1. Meta Newsroom · AI22

    Meta Partners With Five US Universities to Accelerate Medical and Scientific Breakthroughs

    Meta is partnering with five US universities — Caltech, Ohio State's College of Medicine, Louisiana State University, Indiana University and UTHealth Houston — and granting $2 million to support research using its models in medicine, physics, chemistry, biology, computer science and mathematics. Projects include a computer vision system built on Meta's open-weight SAM models to detect tracheostomy tube displacement in infants, multimodal AI to help conversational agents adapt to people with dementia, and open-weight models predicting heart failure risk from routine CT scans and medical records.

October 28Wednesday
  1. Ai2 · Allen Institute for AI40

    How researchers are adapting olmOCR 2 for historical and archaeological archives

    Researchers are adapting Ai2's open olmOCR 2 model to transcribe historical and archaeological archives. Historian Maurice Casey built ArtificeOCR, which runs olmOCR 2 locally with Tropy to make his archival photo library keyword-searchable, while TU Wien's CENTURIA project fine-tuned olmOCR 2 with LoRA on 57 annotated records, cutting transcription error on 450 test records from 15.58% to 1.11%.

  2. Meta AI Research Blog30

    Meta Supports AI Research at Five US Universities to Accelerate Medical and Scientific Breakthroughs

    Meta is supporting research at five US universities — Caltech, Ohio State University College of Medicine, Louisiana State University, Indiana University, and UTHealth Houston — with access to its latest Muse models, researcher collaboration, and funding. Researchers will use Muse Spark Open, Muse Wonder, Muse Spark, the 30B open-weight Muse Glimmer, plus SAM, V-JEPA, DINO, UMA, and SeamlessM4T. Projects span medicine, physics, chemistry, biology, computer science, and mathematics, with universities retaining control of all patient and clinical data.

October 27Tuesday
  1. Apple Newsroom58

    Apple Creator Studio adds AI assistants, MCP support, and MotionVFX in Final Cut Pro

    Apple introduced updates to Apple Creator Studio that bring an intelligent assistant built on Apple and third-party models to Keynote, Pages, and Numbers, plus Model Context Protocol support so external AI agents can read content and perform edits in those apps. Final Cut Pro for Mac now includes built-in MotionVFX and Content Hub stock video, Logic Pro gains a beta assistant and a Stem Splitter that separates drums, and Pixelmator Pro adds Generate Fill for natural-language image edits. The updates arrive on the App Store this November, free for existing subscribers, with new subscriptions at $12.99 per month or $129 per year.

October 24Saturday
  1. Google Research58

    Google Research outlines how its TB screening AI moved from benchmarks to certified clinical deployment

    Google Research described at IDWeek 2026 how its chest X-ray TB screening models progressed from a 2019 Radiology benchmark to a CE MDR Class IIb certified tool deployed by partner Nexus AI in about 40 healthcare facilities across six countries. More than 90 clinicians have evaluated over 25,000 chest radiographs, cutting turnaround from weeks to under 45 seconds, and the tool has also been deployed offline four kilometers underground at South Africa's Mponeng Gold Mine. Google has open-sourced chest X-ray embeddings and MedSigLIP through HAI-DEF and released MedGemma, which it says has passed 10 million downloads, while Indonesia's Ministry of Health is fine-tuning both models on domestic radiographs for a national screening program.

October 23Friday
  1. Amazon Science40

    Audible's Controllable TTS Brings Distinct Character Voices to Audiobooks

    Audible's technology team is using controllable text-to-speech to give audiobook characters distinct, consistent voices, guided by natural-language descriptions of how dialogue should be delivered. A BookLLM component profiles characters and identifies their dialogue with 97% accuracy in English and 94% in Spanish, while a synthetic-data pipeline trains the model on captioned speech. Dialogue differentiation improved by 3.7 points on a MUSHRA test, with statistically significant listening-experience gains for eight of ten voices across English and Spanish tests.

October 22Thursday
  1. Ai2 · Allen Institute for AI58

    Ai2 releases OlmoEarth v1.3 satellite embedding model with global precomputed embeddings and a browser explorer

    Ai2 released OlmoEarth v1.3, a satellite embedding model that turns a year of Sentinel-1, Sentinel-2 and Landsat time series into one 128-dimensional vector per 10-meter pixel, with the first 64 dimensions usable as a smaller embedding. Ai2 precomputed annual embeddings for the whole planet and published them on Source Cooperative, and its 128-dimensional embeddings scored highest on 20 of 27 evaluations against AlphaEarth Foundations and Tessera v2, including a new tropical crop benchmark called PLANTEUR. The release also includes the OlmoEarth Embeddings Explorer, a no-sign-in browser tool for comparing years, finding similar places and training small classifiers on the spot, with 2025, 2024 and 2023 global and 2017 to 2022 covering select countries; weights, pretraining data and code are open.

  2. Google Research58

    Google Research AI model matches expert sonographers on gestational age from blind sweep ultrasound

    Google Research published a prospective multi-center study in JAMA Network Open showing its AI model estimated gestational age from blind sweep ultrasound with a mean absolute error of 4.18 days, versus 4.48 days for expert sonographers using standard biometry. The model was minimally tuned on 120 Chicago participants and tested without any local optimization in Nairobi, where it reached an MAE of 4.27 days, compared with 4.10 days in Chicago. The research targets frontline maternal care, where the WHO estimates more than 250,000 preventable pregnancy-related deaths occur yearly and two-thirds of people lack access to diagnostic imaging.

October 15Thursday
  1. Meta AI Research Blog88

    Meta releases Muse Spark Open, an 816B open-weight multimodal model that runs on two Mac Studios

    Meta released the weights for Muse Spark Open, an open-weight natively multimodal model with 816B total parameters (42B active) and a 1M-token context window, positioned for coding and agentic work. It is the first model in its class to run on a desktop using two Apple M5 Ultra Mac Studios with the MLX framework, and it is available through API partners including Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI, with weights on Hugging Face, Amazon SageMaker AI, Google Cloud Model Garden and Microsoft Foundry Models. Meta says the model ships with auto-tunable Helion GPU kernels in SGLang, delivering 1.03-3.95x kernel-level speedups on NVIDIA and AMD GPUs and a 3% end-to-end speedup on AMD MI350.

    Why it matters: Meta releases open weights for an 816B-parameter multimodal model that runs on two Mac Studios, giving developers a local option for long-horizon coding and agentic tasks.

October 12Monday
  1. Import AI38

    Import AI 476: Creativity and RSI, Worse-Than-China Open Weight Models, and AI in Space

    Import AI 476 covers a new benchmark showing frontier LLMs still trail humans on intuitive visual reasoning, with GPT 6 Astra scoring 53.6 pass@1 versus a 93.1 human baseline. It also discusses AI systems improving at "research taste" via the TasteVal benchmark, and argues AI could accelerate space industrialization and create compounding first-mover advantages.

October 6Tuesday
  1. Mistral AI82

    Mistral launches Mistral Large 4 preview, a 1T-parameter multimodal model with weights due this month

    Mistral has launched a public preview of Mistral Large 4, a 1 trillion-parameter natively multimodal model with 52 billion active parameters, available now through the Mistral Studio API. Mistral says the model is competitive with the strongest open-source models and outperforms other open-weight models from the US and Europe, scoring 82% on an Artificial Analysis Cyber Index vulnerability reproduction and patching test and 49.8% on its combined Coding Agent Index. The weights are set to be released by the end of the month, after red-teaming with cybersecurity partners and authorities; Mistral says the model was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European datacenters.

    Why it matters: Mistral's largest model yet claims open-weight leadership in cybersecurity and agentic coding, with weights promised by month's end.

September 19Saturday
  1. One Useful Thing50

    The Overhang: GPT-6 Astra and Fable 5.1 Show AI's Untapped Capabilities

    Ethan Mollick argues that existing models like GPT-6 Astra and Fable 5.1 can already do weeks of human work when properly guided, yet their capabilities remain barely used. He demonstrates this by having GPT-6 Astra turn the 1977 text adventure Zork into a playable 3D action game and having Fable 5.1 reconstruct Umberto Eco's library in 3D, placing roughly 5,000 identified books among 27,000 shelf slots. Mollick calls this gap the "capability overhang" and says users should pair AI with deep knowledge, wide knowledge, taste, and agency.

September 18Friday
  1. Every42

    Vibe Check: Is GPT-6 Astra a Breakthrough for Indie Filmmakers?

    Creative studio Afterimage tested GPT-6 Astra on footage from real shoots to see whether its Blender 3D skills carried over to visual effects work. Astra handled tasks ranging from replacing a dish in a moving pan to adding a building to a Brooklyn street and making a giant parade float appear outside a window in handheld iPhone footage. Some shots worked and others didn't, but the team ended up dreaming up effects they'd never have attempted before.

August 13Thursday
  1. MiniMax Blog71

    MiniMax releases Music 3.0, an open-weights music generation model for full songs up to five minutes

    MiniMax introduced Music 3.0, an open-weights music generation model that composes, arranges and produces a complete song of up to five minutes from a creative concept and optional lyrics in a single generation. The pipeline pairs an 8B Global LLM initialized from Qwen3.5-8B with a 0.6B Local LLM, fuses their continuous hidden states to condition a 2.4B flow-matching module and a 123M Flow-VAE, and uses eight-layer RVQ plus Structured Captions that describe emotion, instrumentation and vocal delivery over time. MiniMax says this yields more accurate interpretation of creative intent, fuller arrangements and clearer, more natural vocals.

    Why it matters: MiniMax's architecture breakdown shows how an 8B global LLM paired with a 0.6B local model and flow-matching stack targets the long-song coherence problem.

July 31Friday
  1. MiniMax Blog82

    MiniMax launches H3, a general-purpose multimodal generation model with native stereo video at 2K

    MiniMax launched H3, a general-purpose multimodal generation model that understands text, images, video and audio as a unified context and generates video with native stereo sound up to 15 seconds at 2K resolution. The company says 2K output costs less than a third of mainstream models per second, and it plans to open the model weights in the coming days subject to applicable laws and regulations. H3 is built on Contextual Omni Representation, H3-VAE, H3-Omni Transformer and In-Context Regeneration, and MiniMax says a full technical report is coming soon.

    Why it matters: MiniMax's H3 unifies text, image, video and audio generation in one model and claims 2K pricing below a third of mainstream models, with weights promised soon.

July 24Friday
  1. One Useful Thing62

    Ethan Mollick's opinionated guide to which AI to use for real work

    Ethan Mollick published an opinionated guide arguing that using AI now means running agentic systems that combine a model with tools, not just chatting with a chatbot. He recommends ChatGPT (Work and Codex modes) or Claude (Cowork and Code modes) at $20/month for real work, starting with GPT-5.6 Sol or Fable/Opus at High thinking levels, and keeping approval prompts on to guard against prompt injection. He also says Google currently has no leading frontier model or Codex/Code equivalent, though Gemini Notebook and Gemini Omni remain useful for research and video editing.

July 15Wednesday
  1. Modal Blog60

    Thinking Machines' Inkling multimodal model now available on Modal

    Thinking Machines released Inkling, a general-purpose multimodal model that takes text, image and audio inputs and outputs text, and Modal is offering it from day zero as a Managed Endpoint with token-based pricing. Inkling is a mixture-of-experts transformer with 975B total parameters, 41B active, a 1M token context window and native audio and vision, using five sliding-window attention layers for every full-attention layer. On agentic workloads with 8x B200s, Modal's custom DFlash speculator reached 250 tokens per second per user at 2.5M TPM of per-GPU throughput, 67% faster interactivity than the model's built-in speculative path at matched throughput.

    Why it matters: Modal's day-zero endpoint for Thinking Machines' 975B-parameter Inkling shows how a custom DFlash speculator lifts interactivity 67% over the model's built-in speculative path.

June 1Monday
  1. MiniMax Blog87

    MiniMax M3 released with 1M context, frontier coding and native multimodality

    MiniMax released M3, an open-weight model it says is the first to combine frontier coding and agentic performance, a 1M-token context window, and native image and video input. M3 uses MiniMax's new MSA sparse attention architecture, which the company reports cuts per-token compute at 1M context to 1/20 of the previous generation and speeds prefill by more than 9x and decoding by more than 15x. It scores 59.0% on SWE-Bench Pro, 66.0% on Terminal-Bench 2.1 and 74.2% on MCP Atlas, and is available through MiniMax Code, the Token Plan and the API, with weights and a technical report promised within 10 days.

    Why it matters: MiniMax M3 combines a 1M-token context, frontier coding and agentic scores, and native image and video input in one open-weight model, with MSA cutting per-token compute at long context.

May 4Monday
May 23, 2025Friday
  1. Vapi Blog18

    Narakeet: Turn Text Into Natural-Sounding Speech

    Narakeet converts written scripts into lifelike voiceovers, offering 600+ voices across 70+ languages and support for 140+ languages and dialects. The platform syncs narration with PowerPoint, Google Slides, and Markdown files, generates subtitles and captions, and provides a developer API with batch processing. Plans start around $5 per month, with a free tier available.

September 24, 2024Tuesday