DEXTER: Diffusion-Guided EXplanations with TExtual Reasoning for Vision Models

2025/10/16 by Simone Carnemolla, Carnemolla, Simone, Matteo Pennisi +13 · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.m #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2510.14741

arxiv published 2025/10/16 · arxiv updated 2025/11/16

Abstract

Understanding and explaining the behavior of machine learning models is essential for building transparent and trustworthy AI systems. We introduce DEXTER, a data-free framework that employs diffusion models and large language models to generate global, textual explanations of visual classifiers. DEXTER operates by optimizing text prompts to synthesize class-conditional images that strongly activate a target classifier. These synthetic samples are then used to elicit detailed natural language reports that describe class-specific decision patterns and biases. Unlike prior work, DEXTER enables natural language explanation about a classifier's decision process without access to training data or ground-truth labels. We demonstrate DEXTER's flexibility across three tasks-activation maximization, slice discovery and debiasing, and bias explanation-each illustrating its ability to uncover the internal mechanisms of visual classifiers. Quantitative and qualitative evaluations, including a user study, show that DEXTER produces accurate, interpretable outputs. Experiments on ImageNet, Waterbirds, CelebA, and FairFaces confirm that DEXTER outperforms existing approaches in global model explanation and class-level bias reporting. Code is available at https://github.com/perceivelab/dexter.

Citations

• Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models
• Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recency
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Discovering and Mitigating Visual Biases through Keyword Explanation
• Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning
• Exploring CLIP for Assessing the Look and Feel of Images
• Distilling Model Failures as Directions in Latent Space
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings
• REX: Reasoning-aware and Grounded Explanation
• NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks
• Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment
• Salient ImageNet: How to discover spurious features in Deep Learning?
• Just Train Twice: Improving Group Robustness without Training Group\n Information
• How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
• e-ViL: A Dataset and Benchmark for Natural Language Explanations in\n Vision-Language Tasks
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Learning Transferable Visual Models From Natural Language Supervision
• No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
• Estimating Example Difficulty Using Variance of Gradients
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Multimodal Explanations: Justifying Decisions and Pointing to the\n Evidence
• Axiomatic Attribution for Deep Networks
• Towards A Rigorous Science of Interpretable Machine Learning
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Synthesizing the preferred inputs for neurons in neural networks via\n deep generator networks
• Generating Visual Explanations
• Understanding Neural Networks Through Deep Visualization
• Deep Neural Networks are Easily Fooled: High Confidence Predictions for\n Unrecognizable Images
• Deep Learning Face Attributes in the Wild
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers

Discussions

• arxiv.org/abs/2510.14741 오늘도 인상적인 약자를 개발하는 컴퓨터과학자분들입니다 ... [bsky, 4 points, 0 comments]

Related

• Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models
• Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models
• GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
• I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models