Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic\n Circuits

2020/04/13 by Robert Peharz, Peharz, Robert, Steven Lang +15 · 17 citations
Computer Science · #Advanced Neural Network Applications #Bayesian Modeling and Causal Inference #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2004.06231

openalex publication_date 2020/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Probabilistic circuits (PCs) are a promising avenue for probabilistic\nmodeling, as they permit a wide range of exact and efficient inference\nroutines. Recent ``deep-learning-style'' implementations of PCs strive for a\nbetter scalability, but are still difficult to train on real-world data, due to\ntheir sparsely connected computational graphs. In this paper, we propose Einsum\nNetworks (EiNets), a novel implementation design for PCs, improving prior art\nin several regards. At their core, EiNets combine a large number of arithmetic\noperations in a single monolithic einsum-operation, leading to speedups and\nmemory savings of up to two orders of magnitude, in comparison to previous\nimplementations. As an algorithmic contribution, we show that the\nimplementation of Expectation-Maximization (EM) can be simplified for PCs, by\nleveraging automatic differentiation. Furthermore, we demonstrate that EiNets\nscale well to datasets which were previously out of reach, such as SVHN and\nCelebA, and that they can be used as faithful generative image models.\n

Citations

• Auto-Encoding Variational Bayes
• Natural Gradient Works Efficiently in Learning
• Gradient-based learning applied to document recognition
• Sum-Product Networks: A New Deep Architecture
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• How to Square Tensor Networks and Circuits Without Squaring Them
• Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits
• Efficient Probabilistic Tensor Networks
• Distillation of a tractable model from the VQ-VAE
• Approximate Bayesian Inference via Bitstring Representations
• Prediction of Hospital Associated Infections During Continuous Hospital Stays
• Sparse Probabilistic Graph Circuits
• Tractable Sharpness-Aware Learning of Probabilistic Circuits
• Deep Polynomial Chaos Expansion
• Hardware-efficient tractable probabilistic inference for TinyML Neurosymbolic AI applications
• Tractable Representation Learning with Probabilistic Circuits
• Scaling Probabilistic Circuits via Monarch Matrices
• A Quantum Information Theoretic Approach to Tractable Probabilistic Models
• Rethinking Probabilistic Circuit Parameter Learning
• Strudel: Learning Structured-Decomposable Probabilistic Circuits
• Semantic Probabilistic Control of Language Models
• TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation

Related

• Sum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits
• Tractable Representation Learning with Probabilistic Circuits
• Scaling Probabilistic Circuits via Monarch Matrices
• Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits
• Building Expressive and Tractable Probabilistic Generative Models: A Review