Sum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits

2021/09/14 by Zhongjie Yu, Devendra Singh Dhami, Yu, Zhongjie +3
Computer Science · #Explainable Artificial Intelligence (XAI) #Adversarial Robustness in Machine Learning #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2109.06587

Abstract

Probabilistic circuits (PCs) have become the de-facto standard for learning and inference in probabilistic modeling. We introduce Sum-Product-Attention Networks (SPAN), a new generative model that integrates probabilistic circuits with Transformers. SPAN uses self-attention to select the most relevant parts of a probabilistic circuit, here sum-product networks, to improve the modeling capability of the underlying sum-product network. We show that while modeling, SPAN focuses on a specific set of independent assumptions in every product layer of the sum-product network. Our empirical evaluations show that SPAN outperforms state-of-the-art probabilistic generative models on various benchmark data sets as well is an efficient generative image model.

Citations

• Gradient-based learning applied to document recognition

Related

• Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic\n Circuits
• Sum-product networks: A survey
• Sum-Product-Set Networks: Deep Tractable Models for Tree-Structured Graphs
• SPFlow: An Easy and Extensible Library for Deep Probabilistic Learning using Sum-Product Networks
• Tractable Probabilistic Graph Representation Learning with Graph-Induced Sum-Product Networks