Skip to content
Hot eventDeveloping

AWS adds enhanced GPU observability to SageMaker HyperPod

1 reports1 reporting sourcesUpdated 3 h ago

Start here

AI digest

AWS has added enhanced observability to Amazon SageMaker HyperPod, aimed at helping teams diagnose GPU training issues faster on both Amazon EKS and Slurm. A diagnostic summary uses CloudWatch's DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power and SM clock, while static rules surface XID errors, ECC faults, EFA throughput drops and thermal throttling. Default metrics are collected every 30 seconds at roughly 1–2% CPU overhead and are enabled automatically, covering cluster health, CPU/disk/OS-level usage, GPU utilization and temperature, and per-job resource utilization. Advanced metrics are optional because they increase telemetry volume sent to the metric store, which carries a cost. CloudWatch is now the default metric store alongside Amazon Managed Service for Prometheus. The capability is available in all AWS Regions where SageMaker HyperPod is available. In Quick setup, HyperPod observability is enabled by default and cannot be unchecked.

Generated from reports · updated 2 h ago

Report timeline

Follow the reports to see the event from different sides.

November 20
  1. AWS Machine Learning Blog
    Amazon SageMaker HyperPod adds enhanced infrastructure observability for GPU training

    Amazon SageMaker HyperPod now offers enhanced observability to help teams diagnose GPU training issues faster on both Amazon EKS and Slurm. The diagnostic summary uses CloudWatch's DetectGroupOutliers API to flag GPUs deviating on utilization, memory, temperature, power, and SM clock, while static rules surface XID errors, ECC faults, EFA throughput drops, and thermal throttling. Default metrics are collected every 30 seconds at roughly 1–2% CPU overhead, with CloudWatch now the default metric store alongside Amazon Managed Service for Prometheus.

Heat for this event

Not enough continuous observations yet to draw a trend.