MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization

2025/08/11 by A. Jain, Jain, Animesh, Alexandros Stergiou +1 · 1 citation
Computer Science · #Multimodal Machine Learning Applications #Generative Adversarial Networks and Image Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.07833

Abstract

Vision Language Models (VLMs) encode multimodal inputs over large, complex, and difficult-to-interpret architectures, which limit transparency and trust. We propose a Multimodal Inversion for Model Interpretation and Conceptualization (MIMIC) framework that inverts the internal encodings of VLMs. MIMIC uses a joint VLM-based inversion and a feature alignment objective to account for VLM's autoregressive processing. It additionally includes a triplet of regularizers for spatial alignment, natural image smoothness, and semantic realism. We evaluate MIMIC both quantitatively and qualitatively by inverting visual concepts across a range of free-form VLM outputs of varying length. Reported results include both standard visual quality metrics and semantic text-based metrics. To the best of our knowledge, this is the first model inversion approach addressing visual interpretations of VLM concepts.

Citations

• From Colors to Classes: Emergence of Concepts in Vision Transformers
• Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP
• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment
• The Llama 3 Herd of Models
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation
• Interpreting the Second-Order Effects of Neurons in CLIP
• Linear Explanations for Individual Neurons
• SelfIE: Self-Interpretation of Large Language Model Embeddings
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Probing Multimodal Large Language Models for Global and Local Semantic Representations
• FaithLM: Towards Faithful Explanations for Large Language Models
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
• Mistral 7B
• Improved Baselines with Visual Instruction Tuning
• Kronfluence: Influence Functions with Eigenvalue-corrected Kronecker-Factored Approximate Curvature
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Leaping Into Memories: Space-Time Deep Feature Synthesis
• LAION-5B: An open large-scale dataset for training next generation image-text models
• GradViT: Gradient Inversion of Vision Transformers
• Locating and Editing Factual Associations in GPT
• Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Do Vision Transformers See Like Convolutional Neural Networks?
• LoRA: Low-Rank Adaptation of Large Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• The Mind's Eye: Visualizing Class-Agnostic Features of CNNs
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MANGO: A Python Library for Parallel Hyperparameter Tuning
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• Effectively Unbiased FID and Inception Score and where to find them
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Axiomatic Attribution for Deep Networks
• Towards A Rigorous Science of Interpretable Machine Learning
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Improved Techniques for Training GANs
• Learning Deep Features for Discriminative Localization
• Deep Residual Learning for Image Recognition
• Rethinking the Inception Architecture for Computer Vision
• Understanding Neural Networks Through Deep Visualization

Cited by

• TRANSPORTER: Transferring Visual Semantics from VLM Manifolds

Related

• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• Can We Talk Models Into Seeing the World Differently?
• Multi-Modal Interpretability for Enhanced Localization in Vision-Language Models
• Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
• Explaining Multi-modal Large Language Models by Analyzing their Vision Perception