2026/07/18 by Tian Qin, Wei-Min Huang
#stat.ML #cs.LG
Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a 16M-parameter GPT-2-style model reaches validation perplexity 387, compared with 668 for the frozen-trunk control and 2,885 for a matched pure forward-gradient baseline (backpropagation reaches 150). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches 60.5% on CIFAR-10 and 35.2% on CIFAR-100 with a heavy-head design. It reduces peak memory by up to 35% relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.