Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

A PyTorch DDP training loop on Amazon SageMaker HyperPod gains three checks that catch silent data corruption: a non-finite gradient check, a ring replay audit comparing loss bits and gradient fingerprints across nodes, and a weight fingerprint comparison after the optimizer step. Tested with software-injected faults on 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs), the checks name a suspect rank, map it to a node, and let HyperPod replace the node and resume from a checkpoint.