Variational Attention for Sequence-to-Sequence Models

2017/12/21 by Hareesh Bahuleyan, Bahuleyan, Hareesh, Lili Mou +5 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1712.08207

openalex publication_date 2017/12/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The variational encoder-decoder (VED) encodes source information as a set of random variables using a neural network, which in turn is decoded into target data using another neural network. In natural language processing, sequence-to-sequence (Seq2Seq) models typically serve as encoder-decoder networks. When combined with a traditional (deterministic) attention mechanism, the variational latent space may be bypassed by the attention model, and thus becomes ineffective. In this paper, we propose a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables. Results on two experiments show that, without loss of quality, our proposed method alleviates the bypassing phenomenon as it increases the diversity of generated sentences.

Citations

• Auto-Encoding Variational Bayes
• Long Short-Term Memory
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Toward Controlled Generation of Text
• Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models
• Modeling Past and Future for Neural Machine Translation
• Variational Reasoning for Question Answering with Knowledge Graph

Cited by

• STREAM-VAE: Dual-Path Routing for Slow and Fast Dynamics in Vehicle Telemetry Anomaly Detection
• Neurocircuitry-Inspired Hierarchical Graph Causal Attention Networks for Explainable Depression Identification
• Latent Programmer: Discrete Latent Codes for Program Synthesis

Related

• Natural Language Generation with Neural Variational Models
• Prior Attention for Style-aware Sequence-to-Sequence Models
• Discrete Variational Attention Models for Language Generation
• Assessing incrementality in sequence-to-sequence models
• Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning