AWS releases SGLang-Neuron plugin for Trainium inference
Start here
Amazon announced the public availability of SGLang-Neuron, an out-of-tree plugin that runs SGLang on AWS Trainium through an AWS Deep Learning Container, released by RadixArk with Annapurna Labs. Users keep the same SGLang server interface and generation APIs, while the plugin supplies Trainium-specific models, attention, MoE and a graph runner built on native PyTorch and NKI kernels compiled with torch.compile. It also supports PD disaggregation, transferring the prefill KV cache to a decode server via NIXL over EFA, and the walkthrough serves openai/gpt-oss-120b on a Trainium3 EC2 instance with the pytorch-inference-sglang-neuronx container.
From AWS Machine Learning Blog
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogAWS releases SGLang-Neuron, enabling deployment of frontier models on Trainium with the existing SGLang interface
Amazon announced the public availability of SGLang-Neuron, an out-of-tree plugin that runs SGLang on AWS Trainium through an AWS Deep Learning Container, released by RadixArk with Annapurna Labs. Users keep the same SGLang server interface and generation APIs, while the plugin supplies Trainium-specific models, attention, MoE and a graph runner built on native PyTorch and NKI kernels compiled with torch.compile. It also supports PD disaggregation, transferring the prefill KV cache to a decode server via NIXL over EFA, and the walkthrough serves openai/gpt-oss-120b on a Trainium3 EC2 instance with the pytorch-inference-sglang-neuronx container.
Heat for this event
Not enough continuous observations yet to draw a trend.