Interpretable Diffusion Models with B-cos Networks

2025/01/01 by Nicola Bernold, Bernold, Nicola, Moritz Vandenhirtz +5
Computer Science · Mathematics · Physics and Astronomy · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Opinion Dynamics and Social Influence #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2507.03846

openalex publication_date 2025/01/01 · openalex created_date 2025/10/20 · openalex updated_date 2026/07/28

Abstract

Text-to-image diffusion models generate images by iteratively denoising random noise, conditioned on a prompt. While these models have enabled impressive progress in image generation, they often fail to accurately reflect all semantic information described in the prompt -- failures that are difficult to detect automatically. In this work, we introduce a diffusion model architecture built with B-cos modules that offers inherent interpretability. Our approach provides insight into how individual prompt tokens affect the generated image by producing explanations that highlight the pixel regions influenced by each token. We demonstrate that B-cos diffusion models can produce high-quality images while providing meaningful insights into prompt-image alignment.

Citations

• B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable
• B-Cos Aligned Transformers Learn Human-Interpretable Features
• Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation
• Interpretable Diffusion via Information Decomposition
• Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
• B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers
• Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• What the DAAM: Interpreting Stable Diffusion Using Cross Attention
• B-cos Networks: Alignment is All We Need for Interpretability
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Full-Gradient Representation for Neural Network Visualization
• A Unified Approach to Interpreting Model Predictions
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• U-Net: Convolutional Networks for Biomedical Image Segmentation

Related

• Emergence and Evolution of Interpretable Concepts in Diffusion Models
• Patronus: Interpretable Diffusion Models with Prototypes
• Reverse Stable Diffusion: What prompt was used to generate this image?
• Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
• Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation