Skip to content
Zyphra Research·· 12 h agoAI score58

Zyphra scales ZHA subquadratic attention conversion to trillion-parameter models

Scaling Subquadratic Attention to Trillions of Parameters

AI summary

6-Pro, to its Zyphra Hierarchical Attention (ZHA) while keeping the pretrained backbone frozen, and most benchmarks stayed close to the dense baselines. 4T-A95B retained essentially all of its performance, with RULER dropping under two points even at 1M context, and its top-p attention sets grew as a sublinear power law.

Source:Zyphra Research · zyphra.com