Inverting Visual Representations with Convolutional Networks

2015/06/09 by Alexey Dosovitskiy, Thomas Brox, Dosovitskiy, Alexey +1 · 1 voice · 12 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1506.02753

Version 4 - final version to appear in CVPR-2016. Visually better results obtained with feature similarity and adversarial training are in a different paper - arXiv:1602.02644

arxiv published 2015/06/09 · arxiv created 2016/04/26 · arxiv updated 2016/04/26

Abstract

Feature representations, both hand-designed and learned ones, are often hard to analyze and interpret, even when they are extracted from visual data. We propose a new approach to study image representations by inverting them with an up-convolutional neural network. We apply the method to shallow representations (HOG, SIFT, LBP), as well as to deep networks. For shallow representations our approach provides significantly better reconstructions than existing methods, revealing that there is surprisingly rich information contained in these features. Inverting a deep network trained on ImageNet provides several insights into the properties of the feature representation learned by the network. Most strikingly, the colors and the rough contours of an image can be reconstructed from activations in higher network layers and even from the predicted class probabilities.

Citations

• Generating Images with Perceptual Similarity Metrics based on Deep Networks

Cited by

• Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness
• Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
• OpenFLAME: Federated Visual Positioning System to Enable Large-Scale Augmented Reality Applications
• Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics
• Rethinking the Privacy of Text Embeddings: A Reproducibility Study of "Text Embeddings Reveal (Almost) As Much As Text"
• Image Reconstruction as a Tool for Feature Analysis
• Better Language Model Inversion by Compactly Representing Next-Token Distributions
• Does CLIP perceive art the same way we do?
• Securing Transfer-Learned Networks with Reverse Homomorphic Encryption
• Anomaly Detection Based on Critical Paths for Deep Neural Networks
• From Preimage Search To Source-Grounded Feature Inversion
• ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness

Discussions

• Dosovitskiy and Brox's method inverts visual representations with up-convolutional networks, revealing significant detail in both shallow and deep features, reconstructing colors and contours effectiv [bsky, 0 points, 0 comments]

Related

• Understanding Deep Image Representations by Inverting Them
• Every Filter Extracts A Specific Texture In Convolutional Neural Networks
• Interpreting Deep Visual Representations via Network Dissection
• Mechanism of feature learning in convolutional neural networks
• A Powerful Generative Model Using Random Weights for the Deep Image Representation