Google Developers Blog·· 1 d agoAI score58
Google Implements Streaming DiLoCo in MaxText to Enable Cross-Data-Center Distributed LLM Training
Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText
AI summary
Google implemented Streaming DiLoCo in MaxText v0.2.5, its open-source JAX training framework, splitting model parameters into P shards and synchronizing only one shard per step, thereby completing distributed LLM pretraining over low-bandwidth wide-area networks.
Source:Google Developers Blog · developers.googleblog.com