Skip to content
AWS Machine Learning Blog· Roy Allela·· 20 h agoAI score36

Amazon SageMaker HyperPod adds workload recovery and expanded infrastructure checks

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AI summary

Amazon SageMaker HyperPod now offers workload recovery and expanded infrastructure checks to raise training goodput. The additions include MultiNodeTrainingValidation, a deep health check that trains a small model on and across instances to catch NaN or infinite loss, the HyperPod Kubernetes Resource Monitoring Agent for crash-looping device plugins and missing DaemonSets, and hung job detection for Slurm jobs that stall without exiting.

Source:AWS Machine Learning Blog · aws.amazon.com