Zyphra scales ZHA attention conversion to trillion-parameter models
Start here
Zyphra Research reports it has extended its Zyphra Hierarchical Attention (ZHA) conversion method to frontier-scale open-weight models, including Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B and MiMo-V2.6-Pro. The conversion swaps in ZHA subquadratic attention while keeping the pretrained backbone frozen, and Zyphra says most benchmarks stayed close to the dense baselines. For the largest model, Qwen3.8-2.4T-A95B, Zyphra says essentially all performance was retained, with RULER dropping by under two points even at 1M context. It also reports that the model's top-p attention sets grew as a sublinear power law, which the company presents as evidence the approach scales to trillion-parameter models.
Generated from reports · updated 4 h ago
Report timeline
Follow the reports to see the event from different sides.
- Zyphra ResearchZyphra scales ZHA subquadratic attention conversion to trillion-parameter models
6-Pro, to its Zyphra Hierarchical Attention (ZHA) while keeping the pretrained backbone frozen, and most benchmarks stayed close to the dense baselines. 4T-A95B retained essentially all of its performance, with RULER dropping under two points even at 1M context, and its top-p attention sets grew as a sublinear power law.
Heat for this event
- Current, comparable range
- 7
- Peak, comparable range
- 8November 18 05:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.