Zyphra Research·· 12 h agoAI score58
Zyphra scales ZHA subquadratic attention conversion to trillion-parameter models
Scaling Subquadratic Attention to Trillions of Parameters
AI summary
6-Pro, to its Zyphra Hierarchical Attention (ZHA) while keeping the pretrained backbone frozen, and most benchmarks stayed close to the dense baselines. 4T-A95B retained essentially all of its performance, with RULER dropping under two points even at 1M context, and its top-p attention sets grew as a sublinear power law.
Source:Zyphra Research · zyphra.com