Skip to content
AWS Machine Learning Blog· Mijanur Palash·· 3 h agoAI score38

Amazon SageMaker HyperPod adds enhanced infrastructure observability for GPU training

Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

AI summary

Amazon SageMaker HyperPod now offers enhanced observability to help teams diagnose GPU training issues faster on both Amazon EKS and Slurm. The diagnostic summary uses CloudWatch's DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power, and SM clock, while static rules surface XID errors, ECC faults, EFA throughput drops, and thermal throttling. Default metrics are collected every 30 seconds at roughly 1–2% CPU overhead, with CloudWatch now the default metric store alongside Amazon Managed Service for Prometheus.

Source:AWS Machine Learning Blog · aws.amazon.com