A causal framework for explaining the predictions of black-box\n sequence-to-sequence models

2017/07/06 by David Alvarez-Melis, Tommi Jaakkola, Alvarez-Melis, David +1 · 2 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1707.01943

openalex publication_date 2017/07/06 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28

Abstract

We interpret the predictions of any black-box structured input-structured\noutput model around a specific input-output pair. Our method returns an\n"explanation" consisting of groups of input-output tokens that are causally\nrelated. These dependencies are inferred by querying the black-box model with\nperturbed inputs, generating a graph over tokens from the responses, and\nsolving a partitioning problem to select the most relevant components. We focus\nthe general approach on sequence-to-sequence problems, adopting a variational\nautoencoder to yield meaningful input perturbations. We test our method across\nseveral NLP sequence generation tasks.\n

Cited by

• Beyond the Cloud: Assessing the Benefits and Drawbacks of Local LLM Deployment for Translators
• On Measuring Intrinsic Causal Attributions in Deep Neural Networks

Related

• Programs as Black-Box Explanations
• Algorithmic causal structure emerging through compression
• Interpretation of Black Box NLP Models: A Survey
• Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation
• How Transformers Learn Causal Structure with Gradient Descent