Emergent properties with repeated examples

2024/10/09 by François Charton, Julia Kempe, Charton, François +1 · 1 voice · 4 citations
Computer Science · #Advanced Algebra and Logic #Artificial Intelligence (cs.AI) #Constraint Satisfaction and Optimization #FOS: Computer and information sciences #Logic, Reasoning, and Knowledge #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2410.07041

openalex publication_date 2024/10/09 · arxiv published 2024/10/09 · arxiv updated 2024/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

Cited by

• Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Procedural Pretraining: Warming Up Language Models with Abstract Data
• Efficient RL Training for LLMs with Experience Replay

Discussions

• arxiv.org/abs/2410.07041 how did i miss this paper also i know why this works: it is because the repetitions allows you to actually compress the text faithfully, non repeated data is too noisy for yo [bsky, 2 points, 0 comments]

Related

• Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions
• How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
• When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective
• Data Distributional Properties Drive Emergent In-Context Learning in Transformers
• Shared sensitivity to data distribution during learning in humans and transformer networks