Unifying back-propagation and forward-forward algorithms through model predictive control

2024/09/29 by Ren, Lianhai, Li, Qianxiao · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2409.19561

Abstract

We introduce a Model Predictive Control (MPC) framework for training deep neural networks, systematically unifying the Back-Propagation (BP) and Forward-Forward (FF) algorithms. At the same time, it gives rise to a range of intermediate training algorithms with varying look-forward horizons, leading to a performance-efficiency trade-off. We perform a precise analysis of this trade-off on a deep linear network, where the qualitative conclusions carry over to general networks. Based on our analysis, we propose a principled method to choose the optimization horizon based on given objectives and model specifications. Numerical results on various models and tasks demonstrate the versatility of our method.

Cited by

• Learning without Global Backpropagation via Synergistic Information Distillation

Related

• Algorithmic design and implementation considerations of deep MPC
• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Deep Operator Neural Network Model Predictive Control
• Safe and Efficient Model Predictive Control Using Neural Networks: An Interior Point Approach
• Neural Networks for Fast Optimisation in Model Predictive Control: A Review