2018/11/04 by Ahmadreza Ahmadi, Ahmadi, Ahmadreza, Jun Tani +1
Computer Science · #Generative Adversarial Networks and Image Synthesis #Neural Networks and Applications #Anomaly Detection Techniques and Applications
paper · pdf · doi:10.48550/arxiv.1811.01339
This study introduces PV-RNN, a novel variational RNN inspired by the\npredictive-coding ideas. The model learns to extract the probabilistic\nstructures hidden in fluctuating temporal patterns by dynamically changing the\nstochasticity of its latent states. Its architecture attempts to address two\nmajor concerns of variational Bayes RNNs: how can latent variables learn\nmeaningful representations and how can the inference model transfer future\nobservations to the latent variables. PV-RNN does both by introducing adaptive\nvectors mirroring the training data, whose values can then be adapted\ndifferently during evaluation. Moreover, prediction errors during\nbackpropagation, rather than external inputs during the forward computation,\nare used to convey information to the network about the external data. For\ntesting, we introduce error regression for predicting unseen sequences as\ninspired by predictive coding that leverages those mechanisms. The model\nintroduces a weighting parameter, the meta-prior, to balance the optimization\npressure placed on two terms of a lower bound on the marginal likelihood of the\nsequential data. We test the model on two datasets with probabilistic\nstructures and show that with high values of the meta-prior the network\ndevelops deterministic chaos through which the data's randomness is imitated.\nFor low values, the model behaves as a random process. The network performs\nbest on intermediate values, and is able to capture the latent probabilistic\nstructure with good generalization. Analyzing the meta-prior's impact on the\nnetwork allows to precisely study the theoretical value and practical benefits\nof incorporating stochastic dynamics in our model. We demonstrate better\nprediction performance on a robot imitation task with our model using error\nregression compared to a standard variational Bayes model lacking such a\nprocedure.\n