Coherence boosting: When your pretrained language model is not paying enough attention

2021/10/15 by Nikolay Malkin, Zhen Wang, Malkin, Nikolay +3 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2110.08294

openalex publication_date 2021/10/15 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Long-range semantic coherence remains a challenge in automatic language generation and understanding. We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction. We present coherence boosting, an inference procedure that increases a LM's focus on a long context. We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses. It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training.

Cited by

• Short-Context Dominance: How Much Local Context Natural Language Actually Needs?
• Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5
• Mitigating Cross-Image Information Leakage in LVLMs for Multi-Image Tasks

Related

• Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers
• Enhancing elusive clues in knowledge learning by contrasting attention of language models
• Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models
• Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks
• Statistical Coherence Alignment for Large Language Model Representation Learning Through Tensor Field Convergence