Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation

2023/11/09 by Verna Dankers, Ivan Titov, Dankers, Verna +3 · 2 voices
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2311.05379

openalex publication_date 2023/11/09 · openalex created_date 2023/11/11 · openalex updated_date 2026/07/28

Abstract

When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others. Yet, memorisation is not easily expressed as a binary feature that is good or bad: individual datapoints lie on a memorisation-generalisation continuum. What determines a datapoint's position on that spectrum, and how does that spectrum influence neural models' performance? We address these two questions for neural machine translation (NMT) models. We use the counterfactual memorisation metric to (1) build a resource that places 5M NMT datapoints on a memorisation-generalisation map, (2) illustrate how the datapoints' surface-level characteristics and a models' per-datum training signals are predictive of memorisation in NMT, (3) and describe the influence that subsets of that map have on NMT systems' performance.

Discussions

• I was lucky to hear a wonderful talk by @vernadankers.bsky.social today at the IT University of Copenhagen. Verna discussed the relationship between memorization and generalization and I learned so mu [bsky, 38 points, 0 comments]
• What is the term for someone who has similar ideas in parallel to yours, but is a MUCH faster worker, so you're always finding cool recent work and thinking: we should have tried it that way! For me i [bsky, 12 points, 0 comments]

Related

• Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks
• What do larger image classifiers memorise?
• Data Cartography for Detecting Memorization Hotspots and Guiding Data Interventions in Generative Models
• Quantifying Memorization Across Neural Language Models
• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data