Meaning to Form: Measuring Systematicity as Information

2019/06/13 by Tiago Pimentel, Arya D. McCarthy, Pimentel, Tiago +8 · 3 citations
Computer Science · Psychology · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language and cultural evolution #Language, Metaphor, and Cognition #Natural Language Processing Techniques #cs.CL

paper · pdf · doi:10.48550/arxiv.1906.05906

Accepted for publication at ACL 2019

openalex publication_date 2019/06/13 · arxiv created 2019/07/26 · arxiv updated 2019/07/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade? For instance, does the character bigram gl have any systematic relationship to the meaning of words like glisten, gleam and glow? In this work, we offer a holistic quantification of the systematicity of the sign using mutual information and recurrent neural networks. We employ these in a data-driven and massively multilingual approach to the question, examining 106 languages. We find a statistically significant reduction in entropy when modeling a word form conditioned on its semantic representation. Encouragingly, we also recover well-attested English examples of systematic affixes. We conclude with the meta-point: Our approximate effect size (measured in bits) is quite small---despite some amount of systematicity between form and meaning, an arbitrary relationship and its resulting benefits dominate human language.

Citations

• Efficient Estimation of Word Representations in Vector Space
• SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
• Practical Bayesian Optimization of Machine Learning Algorithms
• Arbitrariness, Iconicity, and Systematicity in Language
• Redrawing the margins of language: Lessons from research on ideophones
• An Analysis of Neural Language Modeling at Multiple Scales

Cited by

• Adversarially Probing Cross-Family Sound Symbolism in 27 Languages
• Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
• Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent

Related

• From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
• Construals of iconicity: experimental approaches to form–meaning resemblances in language
• Language Models as Semiotic Machines: Reconceptualizing AI Language Systems through Structuralist and Post-Structuralist Theories of Language
• The link between form and meaning in American Sign Language: Lexical processing effects.
• Signal in Noise: Exploring Meaning Encoded in Random Character Sequences with Character-Aware Language Models