AWS Machine Learning Blog· Roy Allela·· 20 h agoAI score36
Amazon SageMaker HyperPod adds workload recovery and expanded infrastructure checks
Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod
AI summary
Amazon SageMaker HyperPod now offers workload recovery and expanded infrastructure checks to raise training goodput. The additions include MultiNodeTrainingValidation, a deep health check that trains a small model on and across instances to catch NaN or infinite loss, the HyperPod Kubernetes Resource Monitoring Agent for crash-looping device plugins and missing DaemonSets, and hung job detection for Slurm jobs that stall without exiting.
Source:AWS Machine Learning Blog · aws.amazon.com