Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
2025/06/05 by Zesheng Ye, Chengyi Cai, Ye, Zesheng +11 · 1 citation
Computer Science · Engineering · #AI-based Problem Solving and Planning #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2506.04650
openalex publication_date 2025/06/05 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28
Abstract
As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Despite substantial progress, existing adaptation approaches have evolved largely in isolation, without a clear understanding of their interrelationships. This survey introduces neural network reprogrammability as a unifying framework that bridges mainstream model adaptation techniques--model reprogramming, prompt tuning, and prompt instruction--previously fragmented research areas yet converges on a shared principle: repurposing a pre-trained model by manipulating information at the interfaces while keeping the model parameters frozen. These methods exploit neural networks' sensitivity to manipulation on different interfaces, be it through perturbing inputs, inserting tokens into intermediate layers, or providing task-specific examples in context, to redirect model behaviors towards desired outcomes. We then present a taxonomy that categorizes such information manipulation-based adaptation approaches across four key dimensions: manipulation format (fixed or learnable), location (interfaces where manipulations occur), operator (how they are applied), and output alignment requirement (post-processing needed to align outputs with downstream tasks). Notably, this framework applies consistently across data modalities, independent of specific model architectures. Moreover, viewing established techniques like in-context learning and chain-of-thought prompting through this lens reveals both their theoretical connections and practical distinctions. We further analyze remaining technical challenges and ethical considerations, positioning neural network reprogrammability as a fundamental paradigm for efficient model adaptation. We lastly identify promising research directions emerging from this integrative viewpoint.
Citations
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Sample-Specific Noise Injection For Diffusion-Based Adversarial Purification
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
- Model Reprogramming Demystified: A Neural Tangent Kernel Perspective
- One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
- REFINE: Inversion-Free Backdoor Defense via Model Reprogramming
- Attribute-based Visual Reprogramming for Vision-Language Models
- Bayesian-guided Label Mapping for Visual Reprogramming
- Visual Prompting in Multimodal Large Language Models: A Survey
- OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
- A3VLM: Actionable Articulation-Aware Vision Language Model
- Sample-specific Masks for Visual Reprogramming-based Prompting
- Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
- Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
- Model Reprogramming Outperforms Fine-tuning on Out-of-distribution Data in Text-Image Encoders
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
- In-context Learning with Retrieved Demonstrations for Language Models: A Survey
- Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
- AutoVP: An Automated Visual Prompting Framework and Benchmark
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models
- Tuning Multi-mode Token-level Prompt Alignment across Modalities
- Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias
- Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
- Diversity-enhancing Generative Network for Few-shot Hypothesis Adaptation
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- Generating Images with Multimodal Language Models
- Reasoning with Language Model is Planning with World Model
- A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- ImageBind: One Embedding Space To Bind Them All
- Deep Graph Reprogramming
- Visual Instruction Tuning
- What does CLIP know about a red circle? Visual prompt engineering for VLMs
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- Segment Everything Everywhere All at Once
- Micrograph segmentations for DDEVD
- Segment Anything
- Adversarially Contrastive Estimation of Conditional Neural Processes
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Explicit Visual Prompting for Low-Level Structure Segmentations
- GPT-4 Technical Report
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
- UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
- A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models
- What Makes Good Examples for Visual In-Context Learning?
- Reprogramming Pretrained Language Models for Protein Sequence Representation Learning
- Unleashing the Power of Visual Prompting At the Pixel Level
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Structured Prompting: Scaling In-Context Learning to 1,000 Examples
- Images Speak in Images: A Generalist Painter for In-Context Visual Learning
- Language Models as Agent Models
- Understanding and Improving Visual Prompting: A Label-Mapping Perspective
- Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming
- Neural Attentive Circuits
- MaPLe: Multi-modal Prompt Learning
- Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
- Reprogramming Pretrained Language Models for Antibody Sequence Infilling
- Universal Prompt Tuning for Graph Neural Networks
- In-context Learning and Induction Heads
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Visual Prompting via Image Inpainting
- IDNP: Interest Dynamics Modeling using Generative Neural Processes for Sequential Recommendation
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
- Emergent Abilities of Large Language Models
- Adversarial Reprogramming Revisited
- Large Language Models are Zero-Shot Reasoners
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning
- PaLM: Scaling Language Modeling with Pathways
- Exploring Visual Prompts for Adapting Large-Scale Models
- Conditional Prompt Learning for Vision-Language Models
- Contrastive Conditional Neural Processes
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- VL-Adapter: Parameter-Efficient Transfer Learning for\n Vision-and-Language Tasks
- Improving language models by retrieving from trillions of tokens
- Show Your Work: Scratchpads for Intermediate Computation with Language Models
- An Explanation of In-context Learning as Implicit Bayesian Inference
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- CLIP-Adapter: Better Vision-Language Models with Feature Adapters
- CLIP-Adapter: Better Vision-Language Models with Feature Adapters
- Towards a Unified View of Parameter-Efficient Transfer Learning
- PPT: Pre-trained Prompt Tuning for Few-shot Learning
- Learning to Prompt for Vision-Language Models
- Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
- Multimodal Few-Shot Learning with Frozen Language Models
- Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data
- TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation
- PTR: Prompt Tuning with Rules for Text Classification
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions
- The Power of Scale for Parameter-Efficient Prompt Tuning
- Learning Transferable Visual Models From Natural Language Supervision
- Calibrate Before Use: Improving Few-Shot Performance of Language Models
- Cross-modal Adversarial Reprogramming
- WARP: Word-level Adversarial ReProgramming
- Making Pre-trained Language Models Better Few-shot Learners
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Transfer Learning without Knowing: Reprogramming Black-box Machine\n Learning Models with Scarce Data and Limited Resources
- Language Models are Few-Shot Learners
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Adversarial Examples Are Not Bugs, They Are Features
- Parameter-Efficient Transfer Learning for NLP
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- Neural Processes
- Conditional Neural Processes
- Adversarial Reprogramming of Neural Networks
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
- Attention Is All You Need
- Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Understanding deep learning requires rethinking generalization
- Robustness of classifiers: from adversarial to random noise
- Towards Evaluating the Robustness of Neural Networks
- Unsupervised Domain Adaptation with Residual Transfer Networks
- Deep Residual Learning for Image Recognition
- Explaining and Harnessing Adversarial Examples
- Intriguing properties of neural networks
- A Survey on Transfer Learning
- The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
- A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Cited by
Related