Skip to content
Google Developers Blog·· 1 d agoAI score58

Google Implements Streaming DiLoCo in MaxText to Enable Cross-Data-Center Distributed LLM Training

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

AI summary

Google implemented Streaming DiLoCo in MaxText v0.2.5, its open-source JAX training framework, splitting model parameters into P shards and synchronizing only one shard per step, thereby completing distributed LLM pretraining over low-bandwidth wide-area networks.

Source:Google Developers Blog · developers.googleblog.com