Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder
2025/06/25 by Zhang, Yingji, Carvalho, Danilo S., Freitas, André
#Computation and Language (cs.CL) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2506.20083
Abstract
Integrating compositional and symbolic properties into current distributional semantic spaces can enhance the interpretability, controllability, compositionality, and generalisation capabilities of Transformer-based auto-regressive language models (LMs). In this survey, we offer a novel perspective on latent space geometry through the lens of compositional semantics, a direction we refer to as semantic representation learning. This direction enables a bridge between symbolic and distributional semantics, helping to mitigate the gap between them. We review and compare three mainstream autoencoder architectures-Variational AutoEncoder (VAE), Vector Quantised VAE (VQVAE), and Sparse AutoEncoder (SAE)-and examine the distinctive latent geometries they induce in relation to semantic structure and interpretability.
Citations
- The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook
- Base Models Know How to Reason, Thinking Models Learn When
- Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study
- Cross-Layer Discrete Concept Discovery for Interpreting Language Models
- How much do language models memorize?
- Understanding Transformer from the Perspective of Associative Memory
- SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models
- Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering
- I-Con: A Unifying Framework for Representation Learning
- LangVAE and LangSpace: Building and Probing for Language Model VAEs
- A Unified Understanding and Evaluation of Steering Methods
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning
- Improving Steering Vectors by Targeting Sparse Autoencoder Features
- Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering
- ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering
- Exploring Continual Learning of Compositional Generalization in NLI
- TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space
- Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders
- LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces
- Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation
- Steering Llama 2 via Contrastive Activation Addition
- Text Attribute Control via Closed-Loop Disentanglement
- Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders
- Representation Engineering: A Top-Down Approach to AI Transparency
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Steering Language Models With Activation Engineering
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Generating Benchmarks for Factuality Evaluation of Language Models
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- Language Models Implement Simple Word2Vec-style Vector Arithmetic
- A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers
- Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise
- Discovering Language Model Behaviors with Model-Written Evaluations
- Controllable Text Generation via Probability Density Estimation in the Latent Space
- Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder
- A Distributional Lens for Multi-Aspect Controllable Text Generation
- Hyperbolic VAE via Latent Gaussian Distributions
- Learning Disentangled Representations for Natural Language Definitions
- Composable Text Controls in Latent Space with ODEs
- Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Hidden Schema Networks
- Towards Unsupervised Content Disentanglement in Sentence Representations\n via Syntactic Roles
- AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling
- Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs
- Learning Disentangled Representations of Negation and Uncertainty
- Hierarchical Sketch Induction for Paraphrase Generation
- Locating and Editing Factual Associations in GPT
- A Causal Lens for Controllable Text Generation
- VAE based Text Style Transfer with Pivot Words Enhancement Learning
- Disentangling Generative Factors in Natural Language with Discrete\n Variational Autoencoders
- Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
- Entity-Based Knowledge Conflicts in Question Answering
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Factorising Meaning and Form for Intent-Preserving Paraphrasing
- Explaining Answers with Entailment Trees
- Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
- Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors
- Learning Transferable Visual Models From Natural Language Supervision
- GTAE: Graph-Transformer based Auto-Encoders for Linguistic-Constrained Text Style Transfer
- Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs
- Deep Learning for Text Style Transfer: A Survey
- Plug and Play Autoencoders for Conditional Text Generation
- Cycle-Consistent Adversarial Autoencoders for Unsupervised Text Style Transfer
- A Survey on Explainability in Machine Reading Comprehension
- LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
- Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders
- Implicit Deep Latent Variable Models for Text Generation
- Generating Sentences from Disentangled Syntactic and Semantic Spaces
- Syntax-Infused Variational Autoencoder for Text Generation
- Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning
- Unsupervised Paraphrasing without Translation
- Incorporating Sememes into Chinese Definition Modeling
- BERTScore: Evaluating Text Generation with BERT
- Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- Spherical Latent Spaces for Stable Variational Autoencoders
- Disentangled Representation Learning for Non-Parallel Text Style\n Transfer
- Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
- Style Transfer Through Back-Translation
- Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
- Colorless green recurrent networks dream hierarchically
- Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer
- Disentangling by Factorising
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- WorldTree: A Corpus of Explanation Graphs for Elementary Science\n Questions supporting Multi-Hop Inference
- Learning Deep Disentangled Embeddings with the F-Statistic Loss
- Style Transfer in Text: Exploration and Evaluation
- Neural Discrete Representation Learning
- Attention Is All You Need
- Style Transfer from Non-Parallel Text by Cross-Alignment
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
- Sentence Simplification with Deep Reinforcement Learning
- Toward Controlled Generation of Text
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
- Definition Modeling: Learning to define word embeddings in natural\n language
- A Diversity-Promoting Objective Function for Neural Conversation Models
- Character-level Convolutional Networks for Text Classification
- A large annotated corpus for learning natural language inference
- Learning Distributed Word Representations for Natural Logic Reasoning
- Microsoft COCO: Common Objects in Context
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning: A Review and New Perspectives
- Representation Learning: A Review and New Perspectives
- A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts
- Simple Fast Algorithms for the Editing Distance between Trees and Related Problems
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Related