Skip to content
AWS Machine Learning Blog· Summer Devlin·· 19 h agoAI score38

Exa boosts embedding throughput 50% on AWS Inferentia2 and Trainium2 to index billions of documents

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

AI summary

Exa raised embedding throughput by roughly 50% on AWS Neuron hardware (Inferentia2 and Trainium2), reaching about 1.5× its naive FP16 data-parallel baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and attention kernels rewritten for the model's exact shapes to cut padding FLOPS. The work supports re-embedding 30 billion priority documents during a model migration.

Source:AWS Machine Learning Blog · aws.amazon.com