Skip to content
Hot eventDeveloping

Zyphra scales ZHA attention conversion to trillion-parameter models

1 reports1 reporting sourcesUpdated 12 h ago

Start here

AI digest

Zyphra Research reports it has extended its Zyphra Hierarchical Attention (ZHA) conversion method to frontier-scale open-weight models, including Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B and MiMo-V2.6-Pro. The conversion swaps in ZHA subquadratic attention while keeping the pretrained backbone frozen, and Zyphra says most benchmarks stayed close to the dense baselines. For the largest model, Qwen3.8-2.4T-A95B, Zyphra says essentially all performance was retained, with RULER dropping by under two points even at 1M context. It also reports that the model's top-p attention sets grew as a sublinear power law, which the company presents as evidence the approach scales to trillion-parameter models.

Generated from reports · updated 4 h ago

Report timeline

Follow the reports to see the event from different sides.

November 17
  1. Zyphra Research
    Zyphra scales ZHA subquadratic attention conversion to trillion-parameter models

    6-Pro, to its Zyphra Hierarchical Attention (ZHA) while keeping the pretrained backbone frozen, and most benchmarks stayed close to the dense baselines. 4T-A95B retained essentially all of its performance, with RULER dropping under two points even at 1M context, and its top-p attention sets grew as a sublinear power law.

Heat for this event

Current, comparable range
7
Peak, comparable range
8November 18 05:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.