minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models

2022/03/24 by Misra, Kanishka · 9 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2203.13112

Abstract

We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables researchers to apply analysis methods at two levels: (1) at the prediction level -- by providing functions to efficiently extract word/sentence level probabilities; and (2) at the representational level -- by also facilitating efficient extraction of word/phrase level vectors from one or more layers. In this paper, we describe the library and apply it to two motivating case studies: One focusing on the learning dynamics of the BERT architecture on relative grammatical judgments, and the other on benchmarking 23 different LMs on zero-shot abductive reasoning. minicons is available at https://github.com/kanishkamisra/minicons

Cited by

• On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
• Dark & Stormy: Modeling Humor in the Worst Sentences Ever Written
• Hey, wait a minute: on at-issue sensitivity in Language Models
• WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives
• Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models
• Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs
• Is Child-Directed Language Optimized for Word Learning? A Computational Study of Verb Meaning Acquisition
• Quantifier Scope Interpretation in Language Learners and LLMs

Related

• Interactively Providing Explanations for Transformer Language Models
• Small Language Models: Survey, Measurements, and Insights
• Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers
• LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models
• LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models