AWS Machine Learning Blog· Summer Devlin·· 19 h agoAI score38
Exa boosts embedding throughput 50% on AWS Inferentia2 and Trainium2 to index billions of documents
Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips
AI summary
Exa raised embedding throughput by roughly 50% on AWS Neuron hardware (Inferentia2 and Trainium2), reaching about 1.5× its naive FP16 data-parallel baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and attention kernels rewritten for the model's exact shapes to cut padding FLOPS. The work supports re-embedding 30 billion priority documents during a model migration.
Source:AWS Machine Learning Blog · aws.amazon.com