Pre-Trained Image Processing Transformer

2020/12/01 by Hanting Chen, Yunhe Wang, Chen, Hanting +17 · 123 citations
Computer Science · Engineering · #Advanced Image Processing Techniques #Advanced Neural Network Applications #Artificial intelligence #Benchmark (surveying) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Deep learning #Engineering #FOS: Computer and information sciences #Image (mathematics) #Image and Signal Denoising Methods #Image processing #Machine Learning (cs.LG) #Machine learning #Pattern recognition (psychology) #Transformer #Voltage #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.00364

published in arXiv (Cornell University) (Cornell University) · CVPR 2021

openalex publication_date 2020/12/01 · arxiv created 2021/11/08 · arxiv updated 2021/11/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Residual Dense Network for Image Super-Resolution
• Learning Deep Transformer Models for Machine Translation
• A Simple Framework for Contrastive Learning of Visual Representations
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Momentum Contrast for Unsupervised Visual Representation Learning
• End-to-end Interpretable Learning of Non-blind Image Deblurring
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Blur More To Deblur Better: Multi-Blur2Deblur For Efficient Video Deblurring

Cited by

• Low-light Image Enhancement via Multi-scale Attention combined with Fourier Transform
• ScaleResfusion: Residual Rectified Flow based on Residual Vector Field
• SDUM: A Scalable Deep Unrolled Model for Universal MRI Reconstruction
• Differentially Private Feature Release for Wireless Sensing: Adaptive Privacy Budget Allocation on CSI Spectrograms
• Multi-level distortion-aware deformable network for omnidirectional image super-resolution
• Vision-Language Model Guided Image Restoration
• Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolution
• Murmur2Vec: A Hashing Based Solution For Embedding Generation Of COVID-19 Spike Sequences
• FoundIR-v2: Optimizing Pre-Training Data Mixtures for Image Restoration Foundation Model
• Evaluating and Preserving High-level Fidelity in Super-Resolution
• BlurDM: A Blur Diffusion Model for Image Deblurring
• HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
• INSIGHT: An Interpretable Neural Vision-Language Framework for Reasoning of Generative Artifacts
• MRI Super-Resolution with Deep Learning: A Comprehensive Survey
• Semantics and Content Matter: Towards Multi-Prior Hierarchical Mamba for Image Deraining
• CAST-LUT: Tokenizer-Guided HSV Look-Up Tables for Purple Flare Removal
• Hybrid CNN-ViT Framework for Motion-Blurred Scene Text Restoration
• Physics-Informed Image Restoration via Progressive PDE Integration
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning
• Leveraging Learned Image Prior for 3D Gaussian Compression
• SWIR-LightFusion: Multi-spectral Semantic Fusion of Synthetic SWIR with Thermal IR (LWIR/MWIR) and RGB
• Universal Image Restoration Pre-training via Masked Degradation Classification
• High-resolution Photo Enhancement in Real-time: A Laplacian Pyramid Network
• Local-Global Context-Aware and Structure-Preserving Image Super-Resolution
• PyramidStyler: Transformer-Based Neural Style Transfer with Pyramidal Positional Encoding and Reinforcement Learning
• HeadsUp! High-Fidelity Portrait Image Super-Resolution
• LinearSR: Unlocking Linear Attention for Stable and Efficient Image Super-Resolution
• VAMamba: An Efficient Visual Adaptive Mamba for Image Restoration
• Unleashing the Potential of the Semantic Latent Space in Diffusion Models for Image Dehazing
• CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
• What to Remove, What to Preserve: Dual-Ambiguity Rectification for All-in-One Image Restoration
• A review of Recent Techniques for Person Re-Identification
• Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects
• [Re] Improving Interpretation Faithfulness for Vision Transformers
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Time-Aware One Step Diffusion Network for Real-World Image Super-Resolution
• Depth-Aware Super-Resolution via Distance-Adaptive Variational Formulation
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• IA-RED²: Interpretability-Aware Redundancy Reduction for Vision Transformers
• ViTGAN: Training GANs with Vision Transformers
• Augmented Shortcuts for Vision Transformers
• MF-LPR²: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Fourier-Guided Attention Upsampling for Image Super-Resolution
• TFPose: Direct Human Pose Estimation with Transformers
• Rotation Equivariant Arbitrary-scale Image Super-Resolution
• Uncertainty-Aware Spatial Color Correlation for Low-Light Image Enhancement
• Exploiting Diffusion Prior for Task-driven Image Restoration
• Moiré Zero: An Efficient and High-Performance Neural Architecture for Moiré Removal
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• ModalFormer: Multimodal Transformer for Low-Light Image Enhancement
• Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training
• Refiner: Refining Self-attention for Vision Transformers
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking
• VERITAS: Verification and Explanation of Realness in Images for Transparency in AI Systems
• SOLQ: Segmenting Objects by Learning Queries
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• Oneta: Multi-Style Image Enhancement Using Eigentransformation Functions
• Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
• EAMamba: Efficient All-Around Vision State Space Model for Image Restoration
• Visual-Instructed Degradation Diffusion for All-in-One Image Restoration
• Adaptive Blind Super-Resolution Network for Spatial-Specific and Spatial-Agnostic Degradations
• Eformer: Edge Enhancement based Transformer for Medical Image Denoising
• UniRes: Universal Image Restoration for Complex Degradations
• StereoINR: Cross-View Geometry Consistent Stereo Super Resolution with Implicit Neural Representation
• CMT: Convolutional Neural Networks Meet Vision Transformers
• SAAT: Synergistic Alternating Aggregation Transformer for Image Super-Resolution
• One-Step Diffusion-based Real-World Image Super-Resolution with Visual Perception Distillation
• Boosting All-in-One Image Restoration via Self-Improved Privilege Learning
• Optimization of Module Transferability in Single Image Super-Resolution: Universality Assessment and Cycle Residual Blocks
• Efficient RAW Image Deblurring with Adaptive Frequency Modulation
• Dimension-Reduction Attack! Video Generative Models are Experts on Controllable Image Synthesis
• CoSformer: Detecting Co-Salient Object with Transformers
• Multi-View Learning with Context-Guided Receptance for Image Denoising
• PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracy
• Burst Image Super-Resolution via Multi-Cross Attention Encoding and Multi-Scan State-Space Decoding
• Instruct2See: Learning to Remove Any Obstructions Across Distributions
• Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
• PiT: Progressive Diffusion Transformer
• CLIP-aware Domain-Adaptive Super-Resolution
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• DHECA-SuperGaze: Dual Head-Eye Cross-Attention and Super-Resolution for Unconstrained Gaze Estimation
• RealRep: Generalized SDR-to-HDR Conversion via Attribute-Disentangled Representation Learning
• High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution
• Local-to-Global Self-Attention in Vision Transformers
• TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising
• Towards Arbitrary-Scale Spacecraft Image Super-Resolution via Salient Region-Guidance
• Large models for machinery fault diagnosis: Current advances and future directions
• Towards Scale-Aware Low-Light Enhancement via Structure-Guided Transformer Design
• Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing
• NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results
• VL-UR: Vision-Language-guided Universal Restoration of Images Degraded by Adverse Weather Conditions
• Perceptual Image Quality Assessment with Transformers
• DSwinIR: Rethinking Window-based Attention for Image Restoration

Related

• Pre-Trained Image Processing Transformer
• Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation
• On Efficient Transformer-Based Image Pre-training for Low-Level Vision
• Transformers in Vision: A Survey
• How to Train Vision Transformer on Small-scale Datasets?