2023/11/15 by Richard Diehl Martinez, Martinez, Richard Diehl, Zebulon Goriely +12 · 1 voice · 6 citations
Computer Science · Engineering · Mathematics · Psychology · #Artificial intelligence #Baseline (sea) #Computer science #Curriculum #Engineering #Generalization #Language model #Linguistics #Machine learning #Mathematics #Mathematics education #Natural Language Processing Techniques #Natural language processing #Pedagogy #Psychology #Selection (genetic algorithm) #Task (project management) #Text Readability and Simplification #Topic Modeling #Vocabulary #cs.CL
paper · pdf · doi:10.48550/arxiv.2311.08886
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/11/15 · arxiv published 2023/11/15 · arxiv updated 2023/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge. The challenge requires training a language model from scratch using only a relatively small training dataset of ten million words. We experiment with three variants of cognitively-motivated curriculum learning and analyze their effect on the performance of the model on linguistic evaluation tasks. In the vocabulary curriculum, we analyze methods for constraining the vocabulary in the early stages of training to simulate cognitively more plausible learning curves. In the data curriculum experiments, we vary the order of the training instances based on i) infant-inspired expectations and ii) the learning behavior of the model. In the objective curriculum, we explore different variations of combining the conventional masked language modeling task with a more coarse-grained word class prediction task to reinforce linguistic generalization capabilities. Our results did not yield consistent improvements over our own non-curriculum learning baseline across a range of linguistic benchmarks; however, we do find marginal gains on select tasks. Our analysis highlights key takeaways for specific combinations of tasks and settings which benefit from our proposed curricula. We moreover determine that careful selection of model architecture, and training hyper-parameters yield substantial improvements over the default baselines provided by the BabyLM challenge.