Context-dependent representation in recurrent neural networks

2015/06/22 by Gilles Wainrib, Wainrib, Gilles
Biochemistry, Genetics and Molecular Biology · Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Biological sciences #FOS: Physical sciences #Neurons and Cognition (q-bio.NC) #cond-mat.dis-nn #q-bio.NC

paper · pdf · doi:10.48550/arxiv.1506.06602

arxiv created 2016/03/07 · arxiv updated 2016/03/08

Abstract

In order to assess the short-term memory performance of non-linear random neural networks, we introduce a measure to quantify the dependence of a neural representation upon the past context. We study this measure both numerically and theoretically using the mean-field theory for random neural networks, showing the existence of an optimal level of synaptic weights heterogeneity. We further investigate the influence of the network topology, in particular the symmetry of reciprocal synaptic connections, on this measure of context dependence, revealing the importance of considering the interplay between non-linearities and connectivity structure.

Related

• Memory and Information Processing in Recurrent Neural Networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Theory of the superposition principle for randomized connectionist representations in neural networks
• Recurrent neural network models for working memory of continuous variables: activity manifolds, connectivity patterns, and dynamic codes
• Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networks