MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

2022/04/18 by FitzGerald, Jack, Hench, Christopher, Peris, Charith +13 · 19 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2204.08582

Abstract

We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

Cited by

• ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applications
• Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges
• PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
• Intent Clustering with Shared Pseudo-Labels
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data
• TWIST: Training-free and Label-free Short Text Clustering through Iterative Vector Updating with LLMs
• AutoIntent: AutoML for Text Classification
• Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
• Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data
• Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models
• Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
• L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages
• QZhou-Embedding Technical Report
• LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface

Related

• Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
• Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
• Large Language Models: A Survey
• Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact