AWS Machine Learning Blog· Vinay Arora·· 50 min agoAI score58
Amazon SageMaker HyperPod Inference adds managed NVIDIA Dynamo support
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
AI summary
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run multi-node, disaggregated prefill and decode serving with KV-aware routing on vLLM, SGLang, or TensorRT-LLM. AWS reports that disaggregated serving for Qwen3-235B-A22B-FP8 reached 207 versus 101 output tokens per second per GPU compared with aggregated serving, and a two-node Qwen3.5-397B-A17B-FP8 deployment retained 99.1 percent of single-node throughput. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with dynamoSpec, invoked through SageMaker endpoints.
Source:AWS Machine Learning Blog · aws.amazon.com