MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

2022/04/18 by Jack FitzGerald, Christopher Hench, FitzGerald, Jack +31 · 30 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Sentiment Analysis and Opinion Mining #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2204.08582

Preprint; 8 pages

openalex publication_date 2022/04/18 · arxiv created 2022/06/17 · arxiv updated 2022/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

Cited by

• ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applications
• Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges
• PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
• Intent Clustering with Shared Pseudo-Labels
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data
• TWIST: Training-free and Label-free Short Text Clustering through Iterative Vector Updating with LLMs
• AutoIntent: AutoML for Text Classification
• Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
• Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech data
• Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models
• Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
• L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages
• QZhou-Embedding Technical Report
• LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface
• LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
• Effort-Optimized, Accuracy-Driven Labelling and Validation of Test Inputs for DL Systems: A Mixed-Integer Linear Programming Approach
• The Cognate Data Bottleneck in Language Phylogenetics
• KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
• Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
• Survey of Abstract Meaning Representation: Then, Now, Future
• Intent Classification on Low-Resource Languages with Query Similarity Search
• New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models
• What Causes Knowledge Loss in Multilingual Language Models?
• Cequel: Cost-Effective Querying of Large Language Models for Text Clustering
• Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation

Related

• Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
• Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
• Large Language Models: A Survey
• Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact