2021/11/16 by André Ofner, Ofner, André, Sebastian Stober +1
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2111.08792
openalex publication_date 2021/11/16 · openalex created_date 2022/10/24 · openalex updated_date 2026/07/28
We present PredProp, a method for optimization of weights and states in\npredictive coding networks (PCNs) based on the precision of propagated errors\nand neural activity. PredProp jointly addresses inference and learning via\nstochastic gradient descent and adaptively weights parameter updates by\napproximate curvature. Due to the relation between propagated error covariance\nand the Fisher information matrix, PredProp implements approximate Natural\nGradient Descent. We demonstrate PredProp's effectiveness in the context of\ndense decoder networks and simple image benchmark datasets. We found that\nPredProp performs favorably over Adam, a widely used adaptive learning rate\noptimizer in the tested configurations. Furthermore, available optimization\nmethods for weight parameters benefit from using PredProp's error precision\nduring inference. Since hierarchical predictive coding layers are optimised\nindividually using local errors, the required precisions factorize over\nhierarchical layers. Extending beyond classical PCNs with a single set of\ndecoder layers per hierarchical layer, we also generalize PredProp to deep\nneural networks in each PCN layer by additionally factorizing over the weights\nin each PCN layer.\n