A Mutual Information Maximization Perspective of Language Representation Learning

2019/10/18 by Lingpeng Kong, Kong, Lingpeng, Cyprien de Masson d’Autume +9 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1910.08350

openalex publication_date 2019/10/18 · openalex created_date 2019/10/25 · openalex updated_date 2026/07/28

Abstract

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

Citations

• Estimation of Entropy and Mutual Information
• Distributed Representations of Words and Phrases and their Compositionality
• Semi-supervised Sequence Learning
• Learning deep representations by mutual information estimation and maximization
• Learning and Evaluating General Linguistic Intelligence
• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Learning Representations by Maximizing Mutual Information Across Views
• On Mutual Information Maximization for Representation Learning

Cited by

• IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning
• MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
• Decentralized Collaborative Learning Framework for Next POI Recommendation

Related

• Representation Learning with Conditional Information Flow Maximization
• An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understanding
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• A Simple Language Model based on PMI Matrix Approximations
• BURT: BERT-inspired Universal Representation from Learning Meaningful Segment