Error Discovery by Clustering Influence Embeddings

2023/12/07 by Fulton Wang, Julius Adebayo, Wang, Fulton +7 · 2 citations
Computer Science · #Software Testing and Debugging Techniques #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.04712

Abstract

We present a method for identifying groups of test examples -- slices -- on which a model under-performs, a task now known as slice discovery. We formalize coherence -- a requirement that erroneous predictions, within a slice, should be wrong for the same reason -- as a key property that any slice discovery method should satisfy. We then use influence functions to derive a new slice discovery method, InfEmbed, which satisfies coherence by returning slices whose examples are influenced similarly by the training data. InfEmbed is simple, and consists of applying K-Means clustering to a novel representation we deem influence embeddings. We show InfEmbed outperforms current state-of-the-art methods on 2 benchmarks, and is effective for model debugging across several case studies.

Cited by

• LLM generation novelty through the lens of semantic similarity
• Better Training Data Attribution via Better Inverse Hessian-Vector Products

Related

• Active Slice Discovery in Large Language Models
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing
• Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings
• HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging