2023/04/05 by Umais Zahid, Qinghai Guo, Zahid, Umais +3 · 2 citations
Biochemistry, Genetics and Molecular Biology · Engineering · #Advanced Memory and Neural Computing #Artificial Intelligence (cs.AI) #Cell Image Analysis Techniques #FOS: Biological sciences #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Neurons and Cognition (q-bio.NC)
paper · pdf · doi:10.48550/arxiv.2304.02658
openalex publication_date 2023/04/05 · openalex created_date 2023/04/09 · openalex updated_date 2026/07/28
Backpropagation has rapidly become the workhorse credit assignment algorithm for modern deep learning methods. Recently, modified forms of predictive coding (PC), an algorithm with origins in computational neuroscience, have been shown to result in approximately or exactly equal parameter updates to those under backpropagation. Due to this connection, it has been suggested that PC can act as an alternative to backpropagation with desirable properties that may facilitate implementation in neuromorphic systems. Here, we explore these claims using the different contemporary PC variants proposed in the literature. We obtain time complexity bounds for these PC variants which we show are lower-bounded by backpropagation. We also present key properties of these variants that have implications for neurobiological plausibility and their interpretations, particularly from the perspective of standard PC as a variational Bayes algorithm for latent probabilistic models. Our findings shed new light on the connection between the two learning frameworks and suggest that, in its current forms, PC may have more limited potential as a direct replacement of backpropagation than previously envisioned.