2014/12/23 by Maruan Al-Shedivat, Emre Neftci, Al-Shedivat, Maruan +3
Computer Science · Neuroscience · #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Neural dynamics and brain function #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.1412.7272
9 pages, 3 figures, ICLR-15 workshop contribution
openalex publication_date 2014/12/23 · arxiv created 2015/04/22 · arxiv updated 2015/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
The goal of a generative model is to capture the distribution underlying the data, typically through latent variables. After training, these variables are often used as a new representation, more effective than the original features in a variety of learning tasks. However, the representations constructed by contemporary generative models are usually point-wise deterministic mappings from the original feature space. Thus, even with representations robust to class-specific transformations, statistically driven models trained on them would not be able to generalize when the labeled data is scarce. Inspired by the stochasticity of the synaptic connections in the brain, we introduce Energy-based Stochastic Ensembles. These ensembles can learn non-deterministic representations, i.e., mappings from the feature space to a family of distributions in the latent space. These mappings are encoded in a distribution over a (possibly infinite) collection of models. By conditionally sampling models from the ensemble, we obtain multiple representations for every input example and effectively augment the data. We propose an algorithm similar to contrastive divergence for training restricted Boltzmann stochastic ensembles. Finally, we demonstrate the concept of the stochastic representations on a synthetic dataset as well as test them in the one-shot learning scenario on MNIST.