Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs

2026/03/29 by Yi Liu · 1 voice
Computer Science · #cs.AI #cs.PF

paper · pdf · doi:10.48550/arxiv.2603.28823

Abstract

Scaling laws relate model quality to compute budget (FLOPs), but practitioners face wall-clock time constraints, not compute budgets. We study optimal model sizing under fixed time budgets from 5 minutes to 24 hours on consumer GPUs (RTX 4090). Across 70+ runs spanning 50M--1031M parameters, we find: (1)~at each time budget a U-shaped curve emerges where too-small models overfit and too-large models undertrain; (2)~optimal model size follows N^* ∝ t^0.60, growing faster than Chinchilla's N^* ∝ C^0.50, with α= 0.60 ± 0.07 robustly exceeding compute-optimal across all sensitivity analyses; (3)~a dual U-shape mechanism: short-budget U-curves arise from compute bottlenecks, while long-budget U-curves emerge from data bottlenecks (overfitting), with an intermediate regime where the U-curve temporarily disappears. These findings have immediate implications for researchers training on consumer hardware, where wall-clock time -- not FLOPs -- is the binding constraint. We release all code, logs, and 70+ experimental configurations.

Citations

• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Mixtral of Experts
• Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
• Retentive Network: A Successor to Transformer for Large Language Models
• Scaling Data-Constrained Language Models
• RWKV: Reinventing RNNs for the Transformer Era
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training Compute-Optimal Large Language Models
• Unified Scaling Laws for Routed Language Models
• Scaling Laws for Transfer
• Scaling Laws for Neural Language Models
• An Empirical Model of Large-Batch Training

Discussions

• Yi Liu: Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs https://arxiv.org/abs/2603.28823 https://arxiv.org/pdf/2603.28823 https://arxiv.org/html/2603.28823 [bsky, 0 points, 0 comments]

Related

• Time Matters: Scaling Laws for Any Budget
• 100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• How to Rent GPUs on a Budget
• Cramming: Training a Language Model on a Single GPU in One Day