Skip to content
Hot eventDeveloping

SageMaker HyperPod Inference adds managed NVIDIA Dynamo support

1 reports1 reporting sourcesUpdated 43 min ago

Start here

AI digest

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run multi-node, disaggregated prefill and decode serving with KV-aware routing on vLLM, SGLang, or TensorRT-LLM. The HyperPod Inference add-on installs Dynamo's custom resource definitions and serving and scheduling components; vLLM is the default engine, and both single-instance and multi-node deployments are supported, with HyperPod managing node health and replacement. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with dynamoSpec, invoked through SageMaker endpoints. AWS reports that disaggregated serving for Qwen3-235B-A22B-FP8 reached 207 versus 101 output tokens per second per GPU compared with aggregated serving, and that a two-node Qwen3.5-397B-A17B-FP8 deployment retained 99.1 percent of single-node throughput. These figures are AWS's own reported results.

Generated from reports · updated 28 min ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. AWS Machine Learning Blog
    Amazon SageMaker HyperPod Inference adds managed NVIDIA Dynamo support

    Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run multi-node, disaggregated prefill and decode serving with KV-aware routing on vLLM, SGLang, or TensorRT-LLM. AWS reports that disaggregated serving for Qwen3-235B-A22B-FP8 reached 207 versus 101 output tokens per second per GPU compared with aggregated serving, and a two-node Qwen3.5-397B-A17B-FP8 deployment retained 99.1 percent of single-node throughput. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with dynamoSpec, invoked through SageMaker endpoints.

Heat for this event

Not enough continuous observations yet to draw a trend.