Learning the Curriculum with Bayesian Optimization for Task-Specific\n Word Representation Learning

2016/05/12 by Yulia Tsvetkov, Tsvetkov, Yulia, Manaal Faruqui +7 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Algorithms #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1605.03852

openalex publication_date 2016/05/12 · openalex created_date 2022/10/04 · openalex updated_date 2026/07/28

Abstract

We use Bayesian optimization to learn curricula for word representation\nlearning, optimizing performance on downstream tasks that depend on the learned\nrepresentations as features. The curricula are modeled by a linear ranking\nfunction which is the scalar product of a learned weight vector and an\nengineered feature vector that characterizes the different aspects of the\ncomplexity of each instance in the training corpus. We show that learning the\ncurriculum improves performance on a variety of downstream tasks over random\norders and in comparison to the natural corpus order.\n

Cited by

• Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
• Prototype-Guided Curriculum Learning for Zero-Shot Learning

Related

• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• Influence-driven Curriculum Learning for Pre-training on Limited Data
• Learning to Read through Machine Teaching
• Cup Curriculum: Curriculum Learning on Model Capacity
• Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning