Compositional Structures in Neural Embedding and Interaction Decompositions

2024/07/12 by Matthew Trager, Alessandro Achille, Trager, Matthew +7
Computer Science · #Neural Networks and Applications #Cognitive Computing and Networks

paper · pdf · doi:10.48550/arxiv.2407.08934

Abstract

We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks and conditional independence constraints on the probability distributions modeled by these networks. Our framework aims to shed light on the emergence of structural patterns in data representations, a phenomenon widely acknowledged but arguably still lacking a solid formal grounding. Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.

Related

• A Theoretical Analysis of Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition
• Compositionality for Recursive Neural Networks
• When does compositional structure yield compositional generalization? A kernel theory
• Transport of Algebraic Structure to Latent Embeddings
• Compositionality decomposed: how do neural networks generalise?