Skip to content
Hot eventDeveloping

Exa lifts embedding throughput 50% on AWS Neuron chips

1 reports1 reporting sourcesUpdated 19 h ago

Start here

AI digest

Exa says it raised embedding throughput by about 50% on AWS Neuron hardware (Inferentia2 and Trainium2), reaching roughly 1.5× its naive FP16 data-parallel baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and attention kernels rewritten for the model's exact shapes to cut padding FLOPS. The work supports re-embedding 30 billion priority documents during a model migration. Exa runs its workload on a single NeuronCore, packing Kubernetes pods onto larger instances so no cores sit idle. So far it has tuned Inferentia2 only for the US West (Oregon) region, which Exa says limits how far the approach can go. The downcast outputs met Exa's correctness gate of 0.999 cosine similarity against the reference.

Generated from reports · updated 19 h ago

Report timeline

Follow the reports to see the event from different sides.

November 19
  1. AWS Machine Learning Blog
    Exa boosts embedding throughput 50% on AWS Inferentia2 and Trainium2 to index billions of documents

    Exa raised embedding throughput by roughly 50% on AWS Neuron hardware (Inferentia2 and Trainium2), reaching about 1.5× its naive FP16 data-parallel baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and attention kernels rewritten for the model's exact shapes to cut padding FLOPS. The work supports re-embedding 30 billion priority documents during a model migration.

Heat for this event

Current, comparable range
6
Peak, comparable range
10November 19 09:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.