SageMaker HyperPod adds workload recovery and infrastructure checks
Start here
AWS has added workload recovery and expanded infrastructure checks to Amazon SageMaker HyperPod, aiming to catch training failures sooner and raise training goodput. The additions include MultiNodeTrainingValidation, a deep health check that trains a small model on and across instances to catch NaN or infinite loss; the HyperPod Kubernetes Resource Monitoring Agent, which watches the cluster continuously and, when a job fails, reports the pods and nodes involved; and hung job detection for Slurm jobs that stall without exiting. Availability is tied to each tool. MultiNodeTrainingValidation runs when instances are provisioned or on demand. Hung job detection runs as an agent on every node alongside each Slurm job, and AWS says it is off by default. AWS presents the features as a way to catch failures sooner and improve goodput; that is the company's stated aim, not a measured result.
Generated from reports · updated 20 h ago
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogAmazon SageMaker HyperPod adds workload recovery and expanded infrastructure checks
Amazon SageMaker HyperPod now offers workload recovery and expanded infrastructure checks to raise training goodput. The additions include MultiNodeTrainingValidation, a deep health check that trains a small model on and across instances to catch NaN or infinite loss, the HyperPod Kubernetes Resource Monitoring Agent for crash-looping device plugins and missing DaemonSets, and hung job detection for Slurm jobs that stall without exiting.
Heat for this event
- Current, comparable range
- 6
- Peak, comparable range
- 10November 19 08:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.