Native Logical and Hierarchical Representations with Subspace Embeddings

2025/08/21 by Moreira, Gabriel, Marinho, Zita, Marques, Manuel +2
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.16687

Abstract

Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel paradigm: embedding concepts as linear subspaces. This framework inherently models generality via subspace dimensionality and hierarchy through subspace inclusion. It naturally supports set-theoretic operations like intersection (conjunction), linear sum (disjunction) and orthogonal complements (negations), aligning with classical formal semantics. To enable differentiable learning, we propose a smooth relaxation of orthogonal projection operators, allowing for the learning of both subspace orientation and dimension. Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet. Furthermore, on natural language inference benchmarks, our subspace embeddings surpass bi-encoder baselines, offering an interpretable formulation of entailment that is both geometrically grounded and amenable to logical operations.

Citations

• How and where does CLIP process negation?
• The Geometry of Categorical and Hierarchical Concepts in Large Language Models
• Binder: Hierarchical Concept Representation through Order Embedding of Binary Vectors
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
• Multilingual E5 Text Embeddings: A Technical Report
• Shadow Cones: A Generalized Framework for Partial Order Embeddings
• DINOv2: Learning Robust Visual Features without Supervision
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
• ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs
• Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs
• Hierarchical Image Classification using Entailment Cone Embeddings
• VQA-LOL: Visual Question Answering under the Lens of Logic
• Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Hyperbolic Disk Embeddings for Directed Acyclic Graphs
• Representation Learning with Contrastive Predictive Coding
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
• Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures
• Hierarchical Density Order Embeddings
• Representation Tradeoffs for Hyperbolic Embeddings
• Poincaré Embeddings for Learning Hierarchical Representations
• Billion-scale similarity search with GPUs
• Order-Embeddings of Images and Language
• A large annotated corpus for learning natural language inference
• Adam: A Method for Stochastic Optimization
• Word Representations via Gaussian Embedding
• Distributed Representations of Words and Phrases and their Compositionality
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Entity Embeddings with Conceptual Subspaces as a Basis for Plausible Reasoning
• Subspace Representations for Soft Set Operations and Sentence Similarities
• Discrete Word Embedding for Logical Natural Language Understanding
• Learning Distributed Word Representations for Natural Logic Reasoning
• Binder: Hierarchical Concept Representation through Order Embedding of Binary Vectors