AWS guide adds three checks for silent data corruption in SageMaker HyperPod DDP training

AWS published a guide adding three in-loop checks to a PyTorch DistributedDataParallel (DDP) training loop on Amazon SageMaker HyperPod to detect silent data corruption: a non-finite gradient check, a ring replay audit comparing loss bits and gradient fingerprints across nodes, and a weight fingerprint comparison after the optimizer step. In tests with software-injected faults on 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs), the checks identified a suspect rank, mapped it to a node, and let HyperPod replace the node and resume from a checkpoint. The setup requires a HyperPod cluster orchestrated by Amazon EKS with at least three GPU nodes, and checkpoints on shared storage such as FSx for Lustre or S3. AWS notes that replacing a node destroys its root and secondary volumes, including data on local NVMe, and recommends keeping human approval on every replacement until the checks' precision is established on the user's own workload.

Generated from reports · updated 22 h ago

Not enough continuous observations yet to draw a trend.