2020/05/02 by Mengzhou Xia, Antonios Anastasopoulos, Xia, Mengzhou +7 · 6 citations
Computer Science · Mathematics · #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Language model #Machine learning #Mathematics #Natural Language Processing Techniques #Natural language processing #Regression #Statistics #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2005.00870
published in arXiv (Cornell University) (Cornell University) · Accepted at ACL'20
arxiv created 2020/05/02 · openalex publication_date 2020/05/02 · arxiv updated 2020/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting. In this work, we attempt to explore the possibility of gaining plausible judgments of how well an NLP model can perform under an experimental setting, without actually training or testing the model. To do so, we build regression models to predict the evaluation score of an NLP experiment given the experimental settings as input. Experimenting on 9 different NLP tasks, we find that our predictors can produce meaningful predictions over unseen languages and different modeling architectures, outperforming reasonable baselines as well as human experts. Going further, we outline how our predictor can be used to find a small subset of representative experiments that should be run in order to obtain plausible predictions for all other experimental settings.