The science behind TorchNeuron, Amazon's new PyTorch backend for AWS Trainium chips

Amazon released TorchNeuron, an open-source PyTorch backend that translates PyTorch code into operations executable on AWS Trainium systems with minimal or no code changes. It adds an opt-in adaptive eager mode that groups and compiles operations for near-compiled performance, plus multi-level NEFF caching; a shared compilation service cut time to first training step by about a third in a 64-process LLM test. The Annapurna team tuned and benchmarked it using 841 representative models distilled from hundreds of thousands of open-source models.