Task-Based MoE for Multitask Multilingual Machine Translation

2023/08/30 by Hai Pham, Young J. Kim, Pham, Hai +9 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2308.15772

openalex publication_date 2023/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Mixture-of-experts (MoE) architecture has been proven a powerful method for diverse tasks in training deep models in many applications. However, current MoE implementations are task agnostic, treating all tokens from different tasks in the same manner. In this work, we instead design a novel method that incorporates task information into MoE models at different granular levels with shared dynamic task-based adapters. Our experiments and analysis show the advantages of our approaches over the dense and canonical MoE models on multi-task multilingual machine translations. With task-specific adapters, our models can additionally generalize to new tasks efficiently.

Cited by

• X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model

Related

• Scalable and Efficient MoE Training for Multitask Multilingual Models
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
• Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
• Direct Neural Machine Translation with Task-level Mixture of Experts models
• Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters