Direct Preference Optimization for Adaptive Concept-based Explanations

2025/05/21 by Jacopo Teneggi, Zhenzhen Wang, Teneggi, Jacopo +6
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.15626

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Concept-based explanation methods aim at making machine learning models more transparent by finding the most important semantic features of an input (e.g., colors, patterns, shapes) for a given prediction task. However, these methods generally ignore the communicative context of explanations, such as the preferences of a listener. For example, medical doctors understand explanations in terms of clinical markers, but patients may not, needing a different vocabulary to rationalize the same diagnosis. We address this gap with listener-adaptive explanations grounded in principles of pragmatic reasoning and the rational speech act. We introduce an iterative training procedure based on direct preference optimization where a speaker learns to compose explanations that maximize communicative utility for a listener. Our approach only needs access to pairwise preferences, which can be collected from human feedback, making it particularly relevant in real-world scenarios where a model of the listener may not be available. We demonstrate that our method is able to align speakers with the preferences of simulated listeners on image classification across three datasets, and further validate that pragmatic explanations generated with our method improve the classification accuracy of participants in a user study.

Citations

• Because we have LLMs, we Can and Should Pursue Agentic Interpretability
• A Communication-First Account of Explanation
• Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction
• Interactivity x Explainability: Toward Understanding How Interactivity Can Improve Computer Vision Explanations
• Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
• A Survey of Direct Preference Optimization
• Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models
• DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation
• Hypothesis Testing the Circuit Hypothesis in LLMs
• Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalization
• Sufficient and Necessary Explanations (and What Lies in Between)
• The Llama 3 Herd of Models
• Transcoders Find Interpretable LLM Feature Circuits
• DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation
• Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
• Iterative Reasoning Preference Optimization
• The development of human causal learning and reasoning
• Mechanistic Interpretability for AI Safety -- A Review
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
• Gradient based Feature Attribution in Explainable AI: A Technical Review
• Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
• A Survey of Reinforcement Learning from Human Feedback
• Is Ignorance Bliss? The Role of Post Hoc Explanation Faithfulness and Alignment in Model Trust in Laypeople and Domain Experts
• Large Language Model Alignment: A Survey
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Amortizing Pragmatic Program Synthesis with Rankings
• Amortizing Pragmatic Program Synthesis with Rankings
• Pragmatic Inference with a CLIP Listener for Contrastive Captioning
• Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Towards Automated Circuit Discovery for Mechanistic Interpretability
• WiCE: Real-World Entailment for Claims in Wikipedia
• Variational Information Pursuit for Interpretable Predictions
• SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis
• Progress measures for grokking via mechanistic interpretability
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Post-hoc Concept Bottleneck Models
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Training language models to follow instructions with human feedback
• A General Language Assistant as a Laboratory for Alignment
• A Rate-Distortion Framework for Explaining Black-box Model Decisions
• Cartoon Explanations of Image Classifiers
• Machine Learning with a Reject Option: A survey
• A practical introduction to the Rational Speech Act modeling framework
• Learning Transferable Visual Models From Natural Language Supervision
• Explaining by Removing: A Unified Framework for Model Explanation
• Classification with Rejection Based on Cost-sensitive Classification
• Concept Bottleneck Models
• Towards Faithfully Interpretable NLP Systems: How should we define and\n evaluate faithfulness?
• Artificial Intelligence, Values, and Alignment
• Continual adaptation for efficient machine communication
• CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and\n Expert Comparison
• To Trust Or Not To Trust A Classifier
• Pragmatically Informative Image Captioning with Character-Level\n Inference
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Proximal Policy Optimization Algorithms
• Methods for Interpreting and Understanding Deep Neural Networks
• Explanation in Artificial Intelligence: Insights from the Social Sciences
• Explanation in artificial intelligence: Insights from the social sciences
• On Calibration of Modern Neural Networks
• Attention Is All You Need
• A Unified Approach to Interpreting Model Predictions
• Grad-CAM: Why did you say that?
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Rationalizing Neural Predictions
• Cooperative Inverse Reinforcement Learning
• Reasoning About Pragmatics with Neural Listeners and Speakers
• Trust Region Policy Optimization
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Selective Explanations: Leveraging Human Input to Align Explainable AI
• The Explanation Game: Towards Prediction Explainability through Sparse\n Communication
• Directly Optimizing Explanations for Desired Properties
• Diagnostics-Guided Explanation Generation
• Teaching Meaningful Explanations