SageMaker HyperPod Inference adds managed NVIDIA Dynamo support
Start here
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run multi-node, disaggregated prefill and decode serving with KV-aware routing on vLLM, SGLang, or TensorRT-LLM. The HyperPod Inference add-on installs Dynamo's custom resource definitions and serving and scheduling components; vLLM is the default engine, and both single-instance and multi-node deployments are supported, with HyperPod managing node health and replacement. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with dynamoSpec, invoked through SageMaker endpoints. AWS reports that disaggregated serving for Qwen3-235B-A22B-FP8 reached 207 versus 101 output tokens per second per GPU compared with aggregated serving, and that a two-node Qwen3.5-397B-A17B-FP8 deployment retained 99.1 percent of single-node throughput. These figures are AWS's own reported results.
Generated from reports · updated 28 min ago
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogAmazon SageMaker HyperPod Inference adds managed NVIDIA Dynamo support
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run multi-node, disaggregated prefill and decode serving with KV-aware routing on vLLM, SGLang, or TensorRT-LLM. AWS reports that disaggregated serving for Qwen3-235B-A22B-FP8 reached 207 versus 101 output tokens per second per GPU compared with aggregated serving, and a two-node Qwen3.5-397B-A17B-FP8 deployment retained 99.1 percent of single-node throughput. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with dynamoSpec, invoked through SageMaker endpoints.
Heat for this event
Not enough continuous observations yet to draw a trend.