LoRA as a Flexible Framework for Securing Large Vision Systems

2025/05/31 by Blasingame, Zander W., Neddo, Richard E., Liu, Chen
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.00661

Abstract

Adversarial attacks have emerged as a critical threat to autonomous driving systems. These attacks exploit the underlying neural network, allowing small -- nearly invisible -- perturbations to completely alter the behavior of such systems in potentially malicious ways. E.g., causing a traffic sign classification network to misclassify a stop sign as a speed limit sign. Prior working in hardening such systems to adversarial attacks have looked at robust training of the system or adding additional pre-processing steps to the input pipeline. Such solutions either have a hard time generalizing, require knowledge of the adversarial attacks during training, or are computationally undesirable. Instead, we propose to take insights for parameter efficient fine-tuning and use low-rank adaptation (LoRA) to train a lightweight security patch -- enabling us to dynamically patch a large preexisting vision system as new vulnerabilities are discovered. We demonstrate that our framework can patch a pre-trained model to improve classification accuracy by up to 78.01% in the presence of adversarial examples.

Citations

• Data Poisoning in Deep Learning: A Survey
• Fall Leaf Adversarial Attack on Traffic Sign Classification
• Low-Rank Adversarial PGD Attack
• Time Traveling to Defend Against Adversarial Example Attacks in Image Classification
• AdjointDEIS: Efficient Gradients for Diffusion Models
• Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting
• InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
• A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directions
• Multi-LoRA Composition for Image Generation
• Orthogonal Adaptation for Modular Customization of Diffusion Models
• Efficient Vision Transformer for Accurate Traffic Sign Detection
• Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation
• Adversarial Attacks on Traffic Sign Recognition: A Survey
• Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
• Scaling Vision Transformers to 22 Billion Parameters
• RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving
• CurveFormer: 3D Lane Detection by Curve Propagation with Curve Queries and Attention
• BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation
• BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving
• Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon
• Translating Images into Maps
• Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
• LoRA: Low-Rank Adaptation of Large Language Models
• Adversarial Sticker: A Stealthy Attack Method in the Physical World
• End-to-end Lane Shape Prediction with Transformers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Do Better ImageNet Models Transfer Better?
• DARTS: Deceiving Autonomous Cars with Toxic Signs
• Adversarial Patch
• Building Robust Deep Neural Networks for Road Sign Detection
• Adversarial Examples that Fool Detectors
• Robust Physical-World Attacks on Deep Learning Models
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Ensemble Adversarial Training: Attacks and Defenses
• DeepFool: a simple and accurate method to fool deep neural networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Adam: A Method for Stochastic Optimization
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptation
• Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
• StolenLoRA: Exploring LoRA Extraction Attacks via Synthetic Data
• Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Survey
• On the Natural Robustness of Vision-Language Models Against Visual Perception Attacks in Autonomous Driving