Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Paper
•
2501.18512
•
Published
•
29
DiLoCo
-
-
DiLoCo: Distributed Low-Communication Training of Language Models
Paper • 2311.08105 • Published • 15 -
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Paper • 2503.09799 • Published • 13 -
Muon is Scalable for LLM Training
Paper • 2502.16982 • Published -
(Mis)Fitting: A Survey of Scaling Laws
Paper • 2502.18969 • Published -
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Paper • 2112.11446 • Published • 1 -
Scaling Laws for Floating Point Quantization Training
Paper • 2501.02423 • Published • 27 -
Survey on Evaluation of LLM-based Agents
Paper • 2503.16416 • Published • 84