Google Cloud MaxText adds context parallelism for long-context training on TPUs

Google Cloud released long-context training support in MaxText v0.2.4, its open-source JAX training framework, adding four context-parallel strategies on Cloud TPU: All-gather, A2A (DeepSpeed Ulysses), P2P (Ring Attention) and the A2A + P2P hybrid (USP). Benchmarks with Llama 3.1 8B reached 70% MFU on TPU 5p at sequence lengths up to 4 million tokens, and context extended to 14 million tokens on both TPU 5p and TPU 7, with TPU 7 holding 50% MFU below 6 million tokens. The release is available on PyPI, in Docker images and on GitHub main, with sample recipes for TPU 7 and TPU 5p.