MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
2026/04/06 by Zhengqing Yuan, Hanchi Sun, Lichao Sun +1 · 22 voices
#cs.CL #cs.DC #cs.OS
paper · pdf
Abstract
We present MegaTrain, a memory-centric system that efficiently trains 100B+ parameter large language models at full precision on a single GPU. Unlike traditional GPU-centric systems, MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state. To battle the CPU-GPU bandwidth bottleneck, we adopt two key optimizations. 1) We introduce a pipelined double-buffered execution engine that overlaps parameter prefetching, computation, and gradient offloading across multiple CUDA streams, enabling continuous GPU execution. 2) We replace persistent autograd graphs with stateless layer templates, binding weights dynamically as they stream in, eliminating persistent graph metadata while providing flexibility in scheduling. On a single H200 GPU with 1.5TB host memory, MegaTrain reliably trains models up to 120B parameters. It also achieves 1.84× the training throughput of DeepSpeed ZeRO-3 with CPU offloading when training 14B models. MegaTrain also enables 7B model training with 512k token context on a single GH200.
Discussions
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU [hn, 326 points, 57 comments]
- Massive if true arxiv.org/abs/2604.05091 [bsky, 20 points, 2 comments]
- arxiv.org/abs/2604.05091 [bsky, 9 points, 1 comments]
- https://arxiv.org/abs/2604.05091 MegaTrainは、CPUメモリ中心で100B超のLLMを単一GPUで学習できる手法です。 重みと最適化状態をホスト側に置き、必要時だけGPUへ流し込みます。 ダブルバッファと軽量テンプレートで、120B規模や長文学習にも対応します。 [bsky, 1 points, 0 comments]
- 2/2 cómo reducen la demanda de memoria y cómo cualquier startup puede replicar el método. Con 309 puntos y 55 comentarios en Hacker News, la comunidad debate su viabilidad real y costes. Fuente: https [bsky, 0 points, 0 comments]
- Lehigh and Notre Dame team propose MegaTrain, enabling efficient training of large language models on a single GPU by utilizing host memory, enhancing accessibility for AI research. arxiv.org/abs/2604 [bsky, 0 points, 0 comments]
- https://bsky.app/profile/hackernews.com.web.brid.gy/post/3miyqttpbcep2 [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU #HackerNews https://arxiv.org/abs/2604.05091 [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 https://news.ycombinator.com/item?id=47689174 [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 (https://news.ycombinator.com/item?id=47689174) [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU View Article | Join the HN Conversation Summary of HN discussion 🧵👇 [bsky, 0 points, 1 comments]
- 📰 MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU 🔗 https://arxiv.org/abs/2604.05091 💬 Discuss on HN [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 (http://news.ycombinator.com/item?id=47689174) [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 (http://news.ycombinator.com/item?id=47689174) [bsky, 0 points, 0 comments]
- Introducing MegaTrain: a groundbreaking approach that enables full precision training of large language models with over 100 billion parameters on a single GPU. This innovation could revolutionize AI [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU [lemmy, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https:// arxiv.org/abs/2604.05091 # arxiv # llm # llms [mastodon, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 https://news.ycombinator.com/item?id=47689174 [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 (https://news.ycombinator.com/item?id=47689174) [bsky, 0 points, 0 comments]
- MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU https://arxiv.org/abs/2604.05091 (https://news.ycombinator.com/item?id=47689174) [bsky, 0 points, 0 comments]
Related