Pre-Trained Image Processing Transformer

2020/12/01 by Hanting Chen, Yunhe Wang, Chen, Hanting +17 · 56 citations
Computer Science · #Advanced Image Processing Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image and Signal Denoising Methods #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2012.00364

openalex publication_date 2020/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Residual Dense Network for Image Super-Resolution
• Learning Deep Transformer Models for Machine Translation
• A Simple Framework for Contrastive Learning of Visual Representations
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Momentum Contrast for Unsupervised Visual Representation Learning
• End-to-end Interpretable Learning of Non-blind Image Deblurring
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Blur More To Deblur Better: Multi-Blur2Deblur For Efficient Video Deblurring

Cited by

• Low-light Image Enhancement via Multi-scale Attention combined with Fourier Transform
• ScaleResfusion: Residual Rectified Flow based on Residual Vector Field
• SDUM: A Scalable Deep Unrolled Model for Universal MRI Reconstruction
• Differentially Private Feature Release for Wireless Sensing: Adaptive Privacy Budget Allocation on CSI Spectrograms
• Multi-level distortion-aware deformable network for omnidirectional image super-resolution
• Vision-Language Model Guided Image Restoration
• Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolution
• Murmur2Vec: A Hashing Based Solution For Embedding Generation Of COVID-19 Spike Sequences
• FoundIR-v2: Optimizing Pre-Training Data Mixtures for Image Restoration Foundation Model
• Evaluating and Preserving High-level Fidelity in Super-Resolution
• BlurDM: A Blur Diffusion Model for Image Deblurring
• HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
• INSIGHT: An Interpretable Neural Vision-Language Framework for Reasoning of Generative Artifacts
• MRI Super-Resolution with Deep Learning: A Comprehensive Survey
• Semantics and Content Matter: Towards Multi-Prior Hierarchical Mamba for Image Deraining
• CAST-LUT: Tokenizer-Guided HSV Look-Up Tables for Purple Flare Removal
• Hybrid CNN-ViT Framework for Motion-Blurred Scene Text Restoration
• Physics-Informed Image Restoration via Progressive PDE Integration
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning
• Leveraging Learned Image Prior for 3D Gaussian Compression
• SWIR-LightFusion: Multi-spectral Semantic Fusion of Synthetic SWIR with Thermal IR (LWIR/MWIR) and RGB
• Universal Image Restoration Pre-training via Masked Degradation Classification
• High-resolution Photo Enhancement in Real-time: A Laplacian Pyramid Network
• Local-Global Context-Aware and Structure-Preserving Image Super-Resolution
• PyramidStyler: Transformer-Based Neural Style Transfer with Pyramidal Positional Encoding and Reinforcement Learning
• HeadsUp! High-Fidelity Portrait Image Super-Resolution
• LinearSR: Unlocking Linear Attention for Stable and Efficient Image Super-Resolution
• VAMamba: An Efficient Visual Adaptive Mamba for Image Restoration
• Unleashing the Potential of the Semantic Latent Space in Diffusion Models for Image Dehazing
• CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
• What to Remove, What to Preserve: Dual-Ambiguity Rectification for All-in-One Image Restoration
• A review of Recent Techniques for Person Re-Identification
• Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospects
• [Re] Improving Interpretation Faithfulness for Vision Transformers
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Time-Aware One Step Diffusion Network for Real-World Image Super-Resolution
• Depth-Aware Super-Resolution via Distance-Adaptive Variational Formulation
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• IA-RED²: Interpretability-Aware Redundancy Reduction for Vision Transformers
• ViTGAN: Training GANs with Vision Transformers
• Augmented Shortcuts for Vision Transformers
• MF-LPR²: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Fourier-Guided Attention Upsampling for Image Super-Resolution
• TFPose: Direct Human Pose Estimation with Transformers
• Rotation Equivariant Arbitrary-scale Image Super-Resolution
• Uncertainty-Aware Spatial Color Correlation for Low-Light Image Enhancement
• Exploiting Diffusion Prior for Task-driven Image Restoration
• Moiré Zero: An Efficient and High-Performance Neural Architecture for Moiré Removal
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• ModalFormer: Multimodal Transformer for Low-Light Image Enhancement
• Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training
• Refiner: Refining Self-attention for Vision Transformers

Related

• Pre-Trained Image Processing Transformer
• Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation
• On Efficient Transformer-Based Image Pre-training for Low-Level Vision
• Transformers in Vision: A Survey
• How to Train Vision Transformer on Small-scale Datasets?