2022/04/23 by Vishakh Padmakumar, Leonard Lausen, Padmakumar, Vishakh +9 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Stochastic Gradient Optimization Techniques #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2204.11117
NAACL 2022 - Camera ready version
openalex publication_date 2022/04/23 · openalex created_date 2022/04/28 · arxiv created 2022/07/12 · arxiv updated 2022/07/13 · openalex updated_date 2026/07/28
Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.