AWS Machine Learning Blog· Mijanur Palash·· 3 h agoAI score38
Amazon SageMaker HyperPod adds enhanced infrastructure observability for GPU training
Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod
AI summary
Amazon SageMaker HyperPod now offers enhanced observability to help teams diagnose GPU training issues faster on both Amazon EKS and Slurm. The diagnostic summary uses CloudWatch's DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power, and SM clock, while static rules surface XID errors, ECC faults, EFA throughput drops, and thermal throttling. Default metrics are collected every 30 seconds at roughly 1–2% CPU overhead, with CloudWatch now the default metric store alongside Amazon Managed Service for Prometheus.
Source:AWS Machine Learning Blog · aws.amazon.com