Instruction-based image editing: a survey on data, models, evaluation, and applications

2026/03/06 by Xianghao Zang, Zijian Jiang, J. Cheng +9
Arts and Humanities · Computer Science · #Digital Humanities and Scholarship #Generative Adversarial Networks and Image Synthesis #Multimodal Machine Learning Applications #cs.CL #cs.CV

paper · pdf · doi:10.1007/s44336-026-00034-3

openalex publication_date 2026/03/06 · openalex created_date 2026/03/07 · openalex updated_date 2026/07/29

Abstract

Abstract Instruction-based Image Editing (IIE) aims to transform a given image into a new one based on textual instructions. Advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have accelerated progress toward practical “one-sentence image editing” systems. This survey presents a systematic taxonomy and comprehensive review of IIE research, structured around five core dimensions: (1) task definition and hierarchical categorization of editing operations, (2) methodologies for training data construction, (3) architectural evolution from GAN-based to diffusion and autoregressive paradigms, (4) standardized evaluation metrics and benchmark development, and (5) introduction of commercial solutions. Our analysis shows critical technological milestones across model generations. We further propose a Comprehensive, in-Depth, and Diagnostic benchmark for IIE task (CDD-IIE Bench), which can rigorously assess the multiple aspects of model performance. Through empirical comparisons of open-source solutions, we highlight their respective capabilities and limitations. Finally, we discuss future research directions to advance the field.

Citations

• Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
• DreamOmni2: Multimodal Instruction-based Editing and Generation
• X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
• Ovis-U1 Technical Report
• FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
• ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies
• SeedEdit 3.0: Fast and High-Quality Generative Image Editing
• Image Editing As Programs with Diffusion Models
• ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions
• RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions
• HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
• ImgEdit: A Unified Image Editing Dataset and Benchmark
• KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models
• CompBench: Benchmarking Complex Instruction-guided Image Editing
• SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing
• In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer
• Step1X-Edit: A Practical Framework for General Image Editing
• Complex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark
• Anchor Token Matching: Implicit Structure Locking for Training-free AR Image Editing
• Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing
• Training-Free Text-Guided Image Editing with Visual Autoregressive Model
• FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model
• ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing
• Unified Autoregressive Visual Generation and Understanding with Continuous Tokens
• UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer
• Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models
• AnyEdit: Edit Any Knowledge Encoded in Language Models
• REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformations
• EditAR: Unified Conditional Generation with Autoregressive Models
• ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling
• DragScene: Interactive 3D Scene Editing with Single-view Drag Instructions
• BrushEdit: All-In-One Image Inpainting and Editing
• FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models
• Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
• AnyEdit: Mastering Unified High-Quality Image Editing for Any Idea
• OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision
• SeedEdit: Align Image Re-Generation to Image Editing
• DiT4Edit: Diffusion Transformer for Image Editing
• Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing
• Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content
• FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction
• OmniGen: Unified Image Generation
• Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
• I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing
• LLaVA-OneVision: Easy Visual Task Transfer
• SAM 2: Segment Anything in Images and Videos
• UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
• Learning Action and Reasoning-Centric Image Editing from Videos and Simulations
• 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
• ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing
• BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
• Bilateral Reference for High-Resolution Dichotomous Image Segmentation
• Instruct-Imagen: Image Generation with Multi-modal Instruction
• ZONE: Zero-Shot Instruction-Guided Local Editing
• Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
• VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation
• Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation
• SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
• BARET : Balanced Attention based Real image Editing driven by Target-text Inversion
• DreaMo: Articulated 3D Reconstruction From A Single Casual Video
• LEDITS++: Limitless Image Editing using Text-to-Image Models
• Emu Edit: Precise Image Editing via Recognition and Generation Tasks
• The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing
• EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
• Guiding Instruction-based Image Editing via Multimodal Large Language Models
• Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation
• InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
• Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
• MagicEdit: High-Fidelity and Temporally Coherent Video Editing
• CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing
• MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing
• Grounded Text-to-Image Synthesis with Attention Refocusing
• Improving Tuning-Free Real Image Editing with Proximal Guidance
• Recognize Anything: A Strong Image Tagging Model
• InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions
• Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models
• Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
• MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
• Visual Instruction Tuning
• Delta Denoising Score
• An Edit Friendly DDPM Noise Space: Inversion and Manipulations
• Micrograph segmentations for DDEVD
• SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Adding Conditional Control to Text-to-Image Diffusion Models
• Zero-shot Image-to-Image Translation
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
• Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
• EDICT: Exact Diffusion Inversion via Coupled Transformations
• InstructPix2Pix: Learning to Follow Image Editing Instructions
• DiffEdit: Diffusion-based semantic image editing with mask guidance
• UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image
• Imagic: Text-Based Real Image Editing with Diffusion Models
• Prompt-to-Prompt Image Editing with Cross Attention Control
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• RePaint: Inpainting using Denoising Diffusion Probabilistic Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• HairCLIP: Design Your Hair by Text and Reference Image
• CLIPstyler: Image Style Transfer with a Single Text Condition
• Talk-to-Edit: Fine-Grained Facial Editing via Dialog
• StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
• LoRA: Low-Rank Adaptation of Large Language Models
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
• WDNet: Watermark-Decomposition Network for Visible Watermark Removal
• Denoising Diffusion Implicit Models
• PP-OCR: A Practical Ultra Lightweight OCR System
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
• Analyzing and Improving the Image Quality of StyleGAN
• A Style-Based Generator Architecture for Generative Adversarial Networks
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Deep Multi-scale Convolutional Neural Network for Dynamic Scene\n Deblurring
• Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• UIP2P: Unsupervised Instruction-based Image Editing via Edit Reversibility Constraint

Related

• InsightEdit: Towards Better Instruction Following for Image Editing
• Guiding Instruction-based Image Editing via Multimodal Large Language Models
• SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
• CompBench: Benchmarking Complex Instruction-guided Image Editing
• Leveraging LLMs for On-the-Fly Instruction Guided Image Editing