The Power of Scale for Parameter-Efficient Prompt Tuning

2021/04/18 by Brian Lester, Rami Al-Rfou, Lester, Brian +3 · 1 voice · 652 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL

paper · pdf · doi:10.48550/arxiv.2104.08691

Accepted to EMNLP 2021

arxiv published 2021/04/18 · arxiv created 2021/09/02 · arxiv updated 2021/09/03

Abstract

In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's "few-shot" learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.

Cited by

• Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
• FasterPy: An LLM-based Code Execution Efficiency Optimization Framework
• AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• Tokenizing Numerical and Embedding Features for LLM RecSys
• Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias
• Influence of Prompt Engineering on Small Language Models for Guarded Query Routing
• A Systematic Analysis of the Impact of Persona Steering on LLM Capabilities
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
• Hierarchy-Aware Fine-Tuning of Vision-Language Models
• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles
• Latent Implicit Visual Reasoning
• Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictions
• Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection
• Auto-Prompting with Retrieval Guidance for Frame Detection in Logistics
• Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
• Open Ad-hoc Categorization with Contextualized Feature Learning
• Characterizing Mamba's Selective Memory using Auto-Encoders
• FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision
• PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learning
• Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
• Georeferencing complex relative locality descriptions with large language models
• Directional Textual Inversion for Personalized Text-to-Image Generation
• How Prompts Move Language Model Behavior: Frames, Salience, and Construal as Semantic Control
• Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
• Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
• Agile Deliberation: Concept Deliberation for Subjective Visual Classification
• Defect-aware Hybrid Prompt Optimization via Progressive Tuning for Zero-Shot Multi-type Anomaly Detection and Segmentation
• SOP²: Transfer Learning with Scene-Oriented Prompt Pool on 3D Object Detection
• Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
• PVeRA: Probabilistic Vector-Based Random Matrix Adaptation
• LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples
• RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
• Personalized Image Descriptions from Attention Sequences
• Group Orthogonal Low-Rank Adaptation for RGB-T Tracking
• David vs. Goliath: Can Small Models Win Big with Agentic AI in Hardware Design?
• STELLA: Guiding Large Language Models for Time Series Forecasting with Semantic Abstractions
• The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
• MemLoRA: Distilling Expert Adapters for On-Device Memory Systems
• Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time
• Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5
• Multi-Scale Visual Prompting for Lightweight Small-Image Classification
• NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
• Network Self-Configuration based on Fine-Tuned Small Language Models
• PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
• promptolution: A Unified, Modular Framework for Prompt Optimization
• AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy
• ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
• Table as a Modality for Large Language Models
• VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images
• SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models
• Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
• Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptation
• Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
• MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning
• Auxiliary Metrics Help Decoding Skill Neurons in the Wild
• PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
• Profile-LLM: Dynamic Profile Optimization for Realistic Personality Expression in LLMs
• Dual-domain Adaptation Networks for Realistic Image Super-resolution
• Supervised Fine Tuning of Large Language Models for Domain Specific Knowledge Graph Construction:A Case Study on Hunan's Historical Celebrities
• TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning
• ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation
• Dynamic Template Selection for Output Token Generation Optimization: MLP-Based and Transformer Approaches
• MGCA-Net: Multi-Grained Category-Aware Network for Open-Vocabulary Temporal Action Localization
• Backdoor Attacks on Open Vocabulary Object Detectors via Multi-Modal Prompt Tuning
• Medical Knowledge Intervention Prompt Tuning for Medical Image Classification
• CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training
• GenSIaC: Toward Security-Aware Infrastructure-as-Code Generation with Large Language Models
• Selecting Fine-Tuning Examples by Quizzing VLMs
• GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning
• Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition
• Persona-Aware Alignment Framework for Personalized Dialogue Generation
• Advanced Black-Box Tuning of Large Language Models with Limited API Calls
• Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
• Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models
• Remodeling Semantic Relationships in Vision-Language Fine-Tuning
• Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions
• Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
• Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
• Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Models
• OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data
• Order-Level Attention Similarity Across Language Models: A Latent Commonality
• Saliency-Guided Domain Adaptation for Left-Hand Driving in Autonomous Steering
• Plan of Knowledge: Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering
• Adaptive Phase Shift Information Compression for IRS Systems: A Prompt Conditioned Variable Rate Framework
• GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models
• Continual Learning, Not Training: Online Adaptation For Agents
• Keys in the Weights: Transformer Authentication Using Model-Bound Latent Representations
• Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
• Multi-refined Feature Enhanced Sentiment Analysis Using Contextual Instruction
• Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
• Languages are Modalities: Cross-Lingual Alignment via Encoder Injection
• SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models
• Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
• Voice Memory for Agentic Speech Recognition
• Progressive Multimodal Alignment for Continual Instruction Tuning
• ConforNets: Latents-Based Conformational Control in OpenFold3
• Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions
• Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student
• SCOUT: A Lightweight Framework for Scenario Coverage Assessment in Autonomous Driving
• Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation
• LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
• zFLoRA: Zero-Latency Fused Low-Rank Adapters
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Prompts
• FLoRA: Fused forward-backward adapters for parameter efficient fine-tuning and reducing inference-time latencies of LLMs
• Agent-based Automated Claim Matching with Instruction-following LLMs
• OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
• VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
• Network Intrusion Detection: Evolution from Conventional Approaches to LLM Collaboration and Emerging Risks
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study
• Group size effects and collective misalignment in LLM multi-agent systems
• PLAN: Proactive Low-Rank Allocation for Continual Learning
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
• C-NAV: Towards Self-Evolving Continual Object Navigation in Open World
• Latent Space Factorization in LoRA
• COLA: Continual Learning via Autoencoder Retrieval of Adapters
• Restoring Pruned Large Language Models via Lost Component Compensation
• KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
• FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
• ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
• Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
• Prompt-MII: Meta-Learning Instruction Induction for LLMs
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• DEXTER: Diffusion-Guided EXplanations with TExtual Reasoning for Vision Models
• A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
• Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior
• Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity
• Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior
• Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
• State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models
• FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios
• CoSPED: Consistent Soft Prompt Targeted Data Extraction and Defense
• In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
• Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
• X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
• Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge
• Few-shot multi-token DreamBooth with LoRa for style-consistent character generation
• On the Representations of Entities in Auto-regressive Large Language Models
• Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
• Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation
• D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Models
• Vision Language Models: A Survey of 26K Papers
• AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Models
• Upfront Chain-of-Thought: A Cooperative Framework for Chain-of-Thought Compression
• LadderMoE: Ladder-Side Mixture of Experts Adapters for Bronze Inscription Recognition
• Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors
• FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
• SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks
• Neologism Learning for Controllability and Self-Verbalization
• Learning to Rewrite Prompts for Bootstrapping LLMs on Downstream Tasks
• Prompt reinforcing for long-term planning of large language models
• MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
• WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection
• Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning
• FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Method
• Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
• FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuning
• MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts
• LLM Based Bayesian Optimization for Prompt Search
• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
• Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
• Large Language Models Hallucination: A Comprehensive Survey
• SPEAR: Soft Prompt Enhanced Anomaly Recognition for Time Series Data
• Decoupling Task-Solving and Output Formatting in LLM Generation
• Deep Generative Continual Learning using Functional LoRA: FunLoRA
• HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance
• Integrating AI and Ensemble Forecasting: Explainable Materials Planning with Scorecards and Trend Insights for a Large-Scale Manufacturer
• TokMem: Tokenized Procedural Memory for Large Language Models
• Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment
• Retrieval-Augmented Framework for LLM-Based Clinical Decision Support
• Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
• Exploring System 1 and 2 communication for latent reasoning in LLMs
• Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• TAP: Two-Stage Adaptive Personalization of Multi-task and Multi-Modal Foundation Models in Federated Learning
• Zero-Shot Decentralized Federated Learning
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Models
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
• Towards Structured Knowledge: Advancing Triple Extraction from Regional Trade Agreements using Large Language Models
• FedPOB: Sample-Efficient Federated Prompt Optimization via Bandits
• Prompt and Parameter Co-Optimization for Large Language Models
• Pretraining with hierarchical memories: separating long-tail and common knowledge
• GroupCoOp: Group-robust Fine-tuning via Group Prompt Learning
• Temporal Generalization: A Reality Check
• No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization
• Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs
• Memory-Efficient Fine-Tuning via Low-Rank Activation Compression
• F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
• IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
• Context Parametrization with Compositional Adapters
• Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations
• PSRT: Accelerating LRM-based Guard Models via Prefilled Safe Reasoning Traces
• A Tale of Two Experts: Cooperative Learning for Source-Free Unsupervised Domain Adaptation
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
• GALAX: Graph-Augmented Language Model for Explainable Reinforcement-Guided Subgraph Reasoning in Precision Medicine
• Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation
• Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
• Objective-Aligned Direct Answer SFT for Robust Multi-Frame Medical VQA
• Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
• SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
• Riemannian Optimization for LoRA on the Stiefel Manifold
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• TimeMosaic: Temporal Heterogeneity Guided Time Series Forecasting via Adaptive Granularity Patch and Segment-wise Decoding
• HyperAdapt: Simple High-Rank Adaptation
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
• Advances in Large Language Models for Medicine
• Accurate and Efficient Low-Rank Model Merging in Core Space
• Weights-Rotated Preference Optimization for Large Language Models
• K-DeCore: Facilitating Knowledge Transfer in Continual Structured Knowledge Reasoning via Knowledge Decoupling
• Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
• Federated Learning with Ad-hoc Adapter Insertions: The Case of Soft-Embeddings for Training Classifier-as-Retriever
• BEFT: Bias-Efficient Fine-Tuning of Language Models
• Distribution-Aligned Decoding for Efficient LLM Task Adaptation
• Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
• PILOT: Steering Synthetic Data Generation with Psychological & Linguistic Output Targeting
• Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation
• Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identifications
• Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuning
• A Systematic Evaluation of Parameter-Efficient Fine-Tuning Methods for the Security of Code LLMs
• Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
• CBP-Tuning: Efficient Local Customization for Black-box Large Language Models
• NeuroStrike: Neuron-Level Attacks on Aligned LLMs
• POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization
• Context-Aware Language Models for Forecasting Market Impact from Sequences of Financial News
• MAPGD: Multi-Agent Prompt Gradient Descent for Collaborative Prompt Optimization
• PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
• CrunchLLM: Multitask LLMs for Structured Business Reasoning and Outcome Prediction
• Improving Personalized Search with Regularized Low-Rank Parameter Updates
• Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
• CrossPT: Exploring Cross-Task Transferability through Multi-Task Prompt Tuning
• AI-driven Remote Facial Skin Hydration and TEWL Assessment from Selfie Images: A Systematic Solution
• Few-Shot Query Intent Detection via Relation-Aware Prompt Learning
• Pre-Forgettable Models: Prompt Learning as a Native Mechanism for Unlearning
• Enhancing Technical Documents Retrieval for RAG
• Characterizing Fitness Landscape Structures in Prompt Engineering
• Singular Value Few-shot Adaptation of Vision-Language Models
• TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models
• On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility View
• Better by Comparison: Retrieval-Augmented Contrastive Reasoning for Automatic Prompt Optimization
• SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASR
• Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
• ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation
• BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting
• Memory Limitations of Prompt Tuning in Transformers
• Integrating Large Language Models with Network Optimization for Interactive and Explainable Supply Chain Planning: A Real-World Case Study
• A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation
• Towards Instance-wise Personalized Federated Learning via Semi-Implicit Bayesian Prompt Tuning
• ELIXIR: Efficient and LIghtweight model for eXplaIning Recommendations
• Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
• Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data
• ChronoLLM: Customizing Language Models for Physics-Based Simulation Code Generation
• J6: Jacobian-Driven Role Attribution for Multi-Objective Prompt Optimization in LLMs
• STEP: Stepwise Curriculum Learning for Context-Knowledge Fusion in Conversational Recommendation
• SemPT: Semantic Prompt Tuning for Vision-Language Models
• Cross-Prompt Encoder for Low-Performing Languages
• Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Tasks
• Dual Information Speech Language Models for Emotional Conversations
• TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal
• Semantic-Enhanced Time-Series Forecasting via Large Language Models
• AutoAssert 1: A LoRA Fine-Tuned LLM Model for Efficient Automated Assertion Generation
• Score Before You Speak: Improving Persona Consistency in Dialogue Generation using Response Quality Scores
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
• Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning
• PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction
• Optimal Corpus Aware Training for Neural Machine Translation
• Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
• Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models
• Unified modality separation: A vision-language framework for unsupervised domain adaptation
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
• A Foundation Model for DAS Signal Recognition and Visual Prompt Tuning of the Pre-trained Model for Downstream Tasks
• DTPA: Dynamic Token-level Prefix Augmentation for Controllable Text Generation
• Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
• Tensorized Clustered LoRA Merging for Multi-Task Interference
• Efficient Morphology-Aware Policy Transfer to New Embodiments
• EmbedGrad: Gradient-Based Prompt Optimization in Embedding Space for Large Language Models
• Variety Is the Spice of Life: Detecting Misinformation with Dynamic Environmental Representations
• MoKA: Mixture of Kronecker Adapters
• Can LLMs Generate High-Quality Task-Specific Conversations?
• Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
• Intention-Guided Cognitive Reasoning for Egocentric Long-Term Action Anticipation
• Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
• Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models
• Adaptive Content Restriction for Large Language Models via Suffix Optimization
• P3: Prompts Promote Prompting
• RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning
• PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twins
• FaceGCD: Generalized Face Discovery via Dynamic Prefix Generation
• Optimizing Active Learning in Vision-Language Models via Parameter-Efficient Uncertainty Calibration
• Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chess
• Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
• AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation
• Cross-Domain Few-Shot Learning with Coalescent Projections and Latent Space Reservation
• CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation
• ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking
• OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration
• Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator
• LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models
• Hierarchical Cross-modal Prompt Learning for Vision-Language Models
• Privacy Risks of LLM-Empowered Recommender Systems: An Inversion Attack Perspective
• Exploring Scalable Unified Modeling for General Low-Level Vision
• MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio
• Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
• GRID: Scalable Task-Agnostic Prompt-Based Continual Learning for Language Models
• PRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs
• CU-ICU: Customizing Unsupervised Instruction-Finetuned Language Models for ICU Datasets via Text-to-Text Transfer Transformer
• CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
• FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settings
• A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges
• GLAD: Generalizable Tuning for Vision-Language Models
• PROL : Rehearsal Free Continual Learning in Streaming Data via Prompt Online Learning
• On Finetuning Tabular Foundation Models
• Graph Prompting for Graph Learning Models: Recent Advances and Future Directions
• Modeling Code: Is Text All You Need?
• Fusing Large Language Models with Temporal Transformers for Time Series Forecasting
• MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
• Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation
• Foundation Models in Medical Imaging: A Review and Outlook
• Mechanistic Interpretability of LoRA-Adapted Language Models for Nuclear Reactor Safety Applications
• CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning
• Revisiting Pool-based Prompt Learning for Few-shot Class-incremental Learning
• Catastrophic Forgetting Mitigation Through Plateau Phase Activity Profiling
• Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing
• Low-rank Momentum Factorization for Memory Efficient Training
• Defending Against Prompt Injection With a Few DefensiveTokens
• Beyond the Linear Separability Ceiling: Aligning Representations in VLMs
• HGMP:Heterogeneous Graph Multi-Task Prompt Learning
• Visual Instance-aware Prompt Tuning
• SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting
• PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis
• Mask6D: Masked Pose Priors For 6D Object Pose Estimation
• The Primacy of Magnitude in Low-Rank Adaptation
• Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata
• Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation
• Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters
• A Survey on Prompt Tuning
• AdaptaGen: Domain-Specific Image Generation through Hierarchical Semantic Optimization Framework
• Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
• QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation
• PLACE: Prompt Learning for Attributed Community Search in Large Graphs
• Train-before-Test Harmonizes Language Model Rankings
• LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
• Dual Modality-Aware Gated Prompt Tuning for Few-Shot Multimodal Sarcasm Detection
• Context Tuning for In-Context Optimization
• Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection
• PromptSR: Cascade Prompting for Lightweight Image Super-Resolution
• From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEM
• LLM4Hint: Leveraging Large Language Models for Hint Recommendation in Offline Query Optimization
• Open-Vocabulary Object Detection in UAV Imagery: A Review and Future Perspectives
• MemOS: A Memory OS for AI System
• DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning
• EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
• Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
• Activation Reward Models for Few-Shot Model Alignment
• Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
• Knowledge Restoration-driven Prompt Optimization: Unlocking LLM Potential for Open-Domain Relational Triplet Extraction
• Not All Attention Heads Are What You Need: Refining CLIP's Image Representation with Attention Ablation
• Modeling Data Diversity for Joint Instance and Verbalizer Selection in Cold-Start Scenarios
• Impact of Fine-Tuning Methods on Memorization in Large Language Models
• Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
• A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications
• FinStat2SQL: A Text2SQL Pipeline for Financial Statement Analysis
• Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation
• Attention to the Burstiness in Visual Prompt Tuning!
• Prompt Mechanisms in Medical Imaging: A Comprehensive Survey
• Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
• GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles
• Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models
• WaRA: Wavelet Low Rank Adaptation
• Dynamic Context-Aware Prompt Recommendation for Domain-Specific AI Applications
• Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models
• ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIP
• Orthogonal Finetuning Made Scalable
• ConciseHint: Boosting Efficient Reasoning via Continuous Concise Hints during Generation
• Generalizing vision-language models to novel domains: A comprehensive survey
• Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
• NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation
• ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs
• Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
• Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptation
• LoLA-SpecViT: Local Attention SwiGLU Vision Transformer with LoRA for Hyperspectral Imaging
• LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learning
• Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
• Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs
• RiOT: Efficient Prompt Refinement with Residual Optimization Tree
• SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC
• Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts
• Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings
• FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning
• Large Language Models -- the Future of Fundamental Physics?
• Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
• Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systems
• GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
• Theoretically Unmasking Inference Attacks Against LDP-Protected Clients in Federated Vision Models
• Uncertainty-Informed Active Perception for Open Vocabulary Object Goal Navigation
• Empirical Evaluation of Large Language Models in Automated Program Repair
• Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR
• Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning
• Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
• PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence
• Test3R: Learning to Reconstruct 3D at Test Time
• ContextBench: Modifying Contexts for Targeted Latent Activation
• The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
• LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning
• Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
• LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
• E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models
• PE-MA: Parameter-Efficient Co-Evolution of Multi-Agent Systems
• Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation
• Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
• Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
• FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
• Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
• Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning
• Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
• MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
• Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
• Large Language Models are Demonstration Pre-Selectors for Themselves
• Elementary Math Word Problem Generation using Large Language Models
• Cartridges: Lightweight and general-purpose long context representations via self-study
• TALL -- A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
• Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
• Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
• Onboard Optimization and Learning: A Survey
• Communication-Efficient Federated Fine-Tuning
• GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
• How Far Are We from Generating Missing Modalities with Foundation Models?
• Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
• PoLAR: Polar-Decomposed Low-Rank Adapter Representation
• Enhancing Target-unspecific Tasks through a Features Matrix
• Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations
• DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
• AUTOCIRCUIT-RL: Reinforcement Learning-Driven LLM for Automated Circuit Topology Generation
• GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation
• Schema as Parameterized Tools for Universal Information Extraction
• Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification
• PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
• PFMBench: Protein Foundation Model Benchmark
• Prompt-Tuned LLM-Augmented DRL for Dynamic O-RAN Network Slicing
• FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts
• ORAN-GUIDE: RAG-Driven Prompt Learning for LLM-Augmented Reinforcement Learning in O-RAN Network Slicing
• Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
• Beyond Multiple Choice: Evaluating Steering Vectors for Summarization
• Benchmarking Foundation Models for Zero-Shot Biometric Tasks
• Lossless Token Sequence Compression via Meta-Tokens
• Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
• Position: Federated Foundation Language Model Post-Training Should Focus on Open-Source Models
• Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling
• MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
• An Empirical Study of Federated Prompt Learning for Vision Language Model
• Evaluating the Sensitivity of LLMs to Prior Context
• DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers
• EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
• SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
• MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
• MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
• Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
• Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language
• SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
• On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation
• Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models
• Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
• Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning
• Why Do More Experts Fail? A Theoretical Analysis of Model Merging
• Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
• LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
• Exploring the Hidden Capacity of LLMs for One-Step Text Generation
• DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
• Information-Theoretic Complementary Prompts for Improved Continual Text Classification
• PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter
• Improved Representation Steering for Language Models
• Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation
• Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate
• MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning
• CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
• MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning
• UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models
• RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models
• Optimization-Inspired Few-Shot Adaptation for Large Language Models
• RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
• Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection
• ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation
• HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
• LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs
• Learning What to Remember: Test-Time Training via Context Distillation
• Understanding Prompt Tuning and In-Context Learning via Meta-Learning
• Representation Discrepancy Bridging Method for Remote Sensing Image-Text Retrieval
• Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation
• Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
• Align-GRAG: Anchor and Rationale Guided Dual Alignment for Graph Retrieval-Augmented Generation
• Freeze, Prompt, and Adapt: A Framework for Source-free Unsupervised GNN Prompting
• STEAM: A Spatio-TEmporal Alignment Mixture-of-Experts Model with Hierarchical Pre-training for EEG Decoding
• UniErase: Towards Balanced and Precise Unlearning in Language Models
• ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understanding
• CoLA: Collaborative Low-Rank Adaptation
• Small Language Models in the Real World: Insights from Industrial Text Classification
• Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
• GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models
• Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
• The Graph Language: How Knowledge Graphs Speak to Large Language Models
• PRL: Prompts from Reinforcement Learning
• Towards Rehearsal-Free Continual Relation Extraction: Capturing Within-Task Variance with Adaptive Prompting
• EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models
• Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
• OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
• Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities
• Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
• Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems
• Enhancing Latent Computation in Transformers with Latent Tokens
• Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection
• AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
• SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization
• Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering
• Scalable Strategies for Continual Learning with Replay
• SLOT: Sample-specific Language Model Optimization at Test-time
• A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings
• Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform
• Continuous Subspace Optimization for Continual Learning
• Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning
• LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
• Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks
• ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks
• Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild
• Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models
• Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
• ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
• Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment
• Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
• Parallel Scaling Law for Language Models
• MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models
• A Survey on Large Language Models in Multimodal Recommender Systems
• PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
• System Prompt Optimization with Meta-Learning
• Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models
• PrePrompt: Predictive prompting for class incremental learning
• RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models
• FAD: Frequency Adaptation and Diversion for Cross-domain Few-shot Learning
• Evaluating the Effectiveness of Black-Box Prompt Optimization as the Scale of LLMs Continues to Grow
• DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis
• Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
• Injecting Knowledge Graphs into Large Language Models
• DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models
• Causal Prompt Calibration Guided Segment Anything Model for Open-Vocabulary Multi-Entity Segmentation
• Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition
• CacheFL: Privacy-Preserving and Efficient Federated Cache Model Fine-Tuning for Vision-Language Models
• Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
• RASPRef: Retrieval-Augmented Self-Supervised Prompt Refinement for Large Reasoning Models
• Agentic Abstention: Do Agents Know When to Stop Instead of Act?
• PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
• DRIV-EX: Counterfactual Explanations for Driving LLMs
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents
• ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs
• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
• Evolutionary Context Search for Automated Skill Acquisition
• Learning to Translate from Soft to Hard LLM Prompts
• SurfDesign: Effective Protein Design on Molecular Surfaces
• CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
• Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning
• COSMOS: Predictable and Cost-Effective Adaptation of LLMs
• Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision
• TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
• YoChameleon: Personalized Vision and Language Generation
• A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
• CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation
• Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?
• Latent Personal Memory: Represent personal memory as dynamic soft prompts
• Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
• Low-Rank Adaptation Redux for Large Models
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
• Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
• Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse
• E-InMeMo: Enhanced Prompting for Visual In-Context Learning
• Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding
• On the Role of Computation in Reinforcement Learning
• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
• OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet
• LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment
• HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
• A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation
• Robo-Troj: Attacking LLM-based Task Planners
• Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO
• When Do PEFT Adaptations Leak Structure? Measuring Black-Box Structural Bounds in Public-Base Model Services
• Can LLMs Interpret and Leverage Structured Linguistic Representations? A Case Study with AMRs
• Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting
• SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for Unpaired RGB+Thermal 3D Reconstruction
• Prompt-Tuning SAM: From Generalist to Specialist with only 2048 Parameters and 16 Training Images
• Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
• CAPO: Cost-Aware Prompt Optimization
• DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation
• AROMA: Autonomous Rank-one Matrix Adaptation
• Large models for machinery fault diagnosis: Current advances and future directions
• PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning
• Few-shot Hate Speech Detection Based on the MindSpore Framework
• Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability
• Histogram-based Parameter-efficient Tuning for Passive and Active Sonar Classification
• PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines
• Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation
• CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey
• Visual Consensus Prompting for Co-Salient Object Detection
• Bayesian Principles Improve Prompt Learning In Vision-Language Models
• Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
• Controlled Territory and Conflict Tracking (CONTACT): (Geo-)Mapping Occupied Territory from Open Source Intelligence
• LoRA-Based Continual Learning with Constraints on Critical Parameter Changes
• The Future of Internet of Things and Multimodal Language Models in 6G Networks: Opportunities and Challenges
• Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions
• Investigating and Mitigating Stereotype-aware Unfairness in LLM-based Recommendations
• FiGO: Fine-Grained Object Counting without Annotations
• DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency
• Beyond Full-Model Rollback: AuroSFT for Adapter-State Multi-Task Fine-Tuning
• KV-Skill: Forging Expertise in the Model's Native Language
• Adapting Vision Foundation Models with Cascaded Semantics
• PolyAlign: Conditional Human-Distribution Alignment
• Skill Neologisms: Towards Skill-based Continual Learning
• DMPT: Decoupled Modality-aware Prompt Tuning for Multi-modal Object Re-identification
• UP-Person: Unified Parameter-Efficient Transfer Learning for Text-based Person Retrieval
• A Survey on Efficient Vision-Language Models
• C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing
• Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation
• Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction
• LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
• DUKAE: DUal-level Knowledge Accumulation and Ensemble for Pre-Trained Model-Based Continual Learning
• Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer
• ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models
• Automated Business Process Analysis: An LLM-Based Approach to Value Assessment
• Leveraging Prompt-Tuning for Bengali Grammatical Error Explanation Using Large Language Models
• SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understanding
• PF3Det: A Prompted Foundation Feature Assisted Visual LiDAR 3D Detector
• SpectR: Dynamically Composing LM Experts with Spectral Routing
• GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization

Discussions

• “Prompt tuning” closes the gap with fine tuning in large language models [hn, 1 points, 0 comments]

Related

• XPrompt: Exploring the Extreme of Prompt Tuning
• FPT: Improving Prompt Tuning Efficiency via Progressive Training
• DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
• Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
• PPT: Pre-trained Prompt Tuning for Few-shot Learning