Neural Language Priors

2019/10/04 by Joseph Enguehard, Enguehard, Joseph, Dan Busbridge +5
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language and cultural evolution #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1910.03492

openalex publication_date 2019/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The choice of sentence encoder architecture reflects assumptions about how a sentence's meaning is composed from its constituent words. We examine the contribution of these architectures by holding them randomly initialised and fixed, effectively treating them as as hand-crafted language priors, and evaluating the resulting sentence encoders on downstream language tasks. We find that even when encoders are presented with additional information that can be used to solve tasks, the corresponding priors do not leverage this information, except in an isolated case. We also find that apparently uninformative priors are just as good as seemingly informative priors on almost all tasks, indicating that learning is a necessary component to leverage information provided by architecture choice.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Networks and the Bias/Variance Dilemma
• A vector space model for automatic indexing
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Distributional Structure
• Deep Residual Learning for Image Recognition

Related

• Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models
• Towards a neural architecture of language: Deep learning versus logistics of access in neural architectures for compositional processing
• Can neural networks acquire a structural bias from raw linguistic data?
• Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data
• Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)