Towards Evaluating the Robustness of Neural Networks

2016/08/16 by Carlini, Nicholas, Wagner, David · 184 citations
#Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1608.04644

Abstract

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x' that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%. In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

Cited by

• Towards Reliable Evaluation of Adversarial Robustness for Spiking Neural Networks
• SemCovert: Secure and Covert Video Transmission via Deep Semantic-Level Hiding
• Game of Coding under Computation-Dependent Adversarial Noise
• A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection
• Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
• Defending against adversarial attacks using mixture of experts
• Stabilizing Multimodal Autoencoders: A Theoretical and Empirical Analysis of Fusion Strategies
• Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications
• SafeMed-R1: Adversarial Reinforcement Learning for Generalizable and Robust Medical Reasoning in Vision-Language Models
• Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
• Enhancing Decision-Making in Windows PE Malware Classification During Dataset Shifts with Uncertainty Estimation
• Adversarial Robustness of Vision in Open Foundation Models
• Adversarial VR: An Open-Source Testbed for Evaluating Adversarial Robustness of VR Cybersickness Detection and Mitigation
• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
• Quantitative Verification of Fairness in Tree Ensembles
• Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance
• Autoencoder-based Denoising Defense against Adversarial Attacks on Object Detection
• When sufficiency is insufficient: the functional information bottleneck for identifying probabilistic neural representations
• Optimizing the Adversarial Perturbation with a Momentum-based Adaptive Matrix
• Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries
• Behavior-Aware and Generalizable Defense Against Black-Box Adversarial Attacks for ML-Based IDS
• PHANTOM: PHysical ANamorphic Threats Obstructing Connected Vehicle Mobility
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• Bilevel Optimization for Covert Memory Tampering in Heterogeneous Multi-Agent Architectures (XAMT)
• Empirical evaluation of the Frank-Wolfe methods for constructing white-box adversarial attacks
• Learning to Split: A Reinforcement-Learning-Guided Splitting Heuristic for Neural Network Verification
• Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation
• Forecasting Fails: Unveiling Evasion Attacks in Weather Prediction Models
• Towards Robust DeepFake Detection under Unstable Face Sequences: Adaptive Sparse Graph Embedding with Order-Free Representation and Explicit Laplacian Spectral Prior
• Rethinking Robustness: A New Approach to Evaluating Feature Attribution Methods
• SPOOF: Simple Pixel Operations for Out-of-Distribution Fooling
• Adversarial Limits of Quantum Certification: When Eve Defeats Detection
• Automatic Attack Discovery for Few-Shot Class-Incremental Learning via Large Language Models
• Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustness
• FeatureLens: A Highly Generalizable and Interpretable Framework for Detecting Adversarial Examples Based on Image Features
• Defense That Attacks: How Robust Models Become Better Attackers
• QSTAformer: A Quantum-Enhanced Transformer for Robust Short-Term Voltage Stability Assessment against Adversarial Attacks
• Superpixel Attack: Enhancing Black-box Adversarial Attack with Image-driven Division Areas
• INSIGHT: An Interpretable Neural Vision-Language Framework for Reasoning of Generative Artifacts
• Towards Trustworthy Wi-Fi CSI-based Sensing: Systematic Evaluation of Adversarial Robustness
• Frequency Bias Matters: Diving into Robust and Generalized Deep Image Forgery Detection
• AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents
• Robust Physical Adversarial Patches Using Dynamically Optimized Clusters
• Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
• A Novel and Practical Universal Adversarial Perturbations against Deep Reinforcement Learning based Intrusion Detection Systems
• ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
• Enhancing Adversarial Transferability through Block Stretch and Shrink
• Layer-wise Noise Guided Selective Wavelet Reconstruction for Robust Medical Image Segmentation
• TopoReformer: Mitigating Adversarial Attacks Using Topological Purification in OCR Models
• HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation
• Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation
• Transferable Dual-Domain Feature Importance Attack against AI-Generated Image Detector
• Power Homotopy for Zeroth-Order Non-Convex Optimizations
• Shedding Light on VLN Robustness: A Black-box Framework for Indoor Lighting-based Adversarial Attack
• MPD-SGR: Robust Spiking Neural Networks with Membrane Potential Distribution-Driven Surrogate Gradient Regularization
• Cybersecurity of High-Altitude Platform Stations: Threat Taxonomy, Attacks and Defenses with Standards Mapping - DDoS Attack Use Case
• Dynamic Parameter Optimization for Highly Transferable Transformation-Based Attacks
• Unsupervised Robust Domain Adaptation: Paradigm, Theory and Algorithm
• MiniFool -- Physics-Constraint-Aware Minimizer-Based Adversarial Attacks in Deep Neural Networks
• A Generative Adversarial Approach to Adversarial Attacks Guided by Contrastive Language-Image Pre-trained Model
• Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
• 3D-ANC: Adaptive Neural Collapse for Robust 3D Point Cloud Recognition
• Probably Approximately Global Robustness Certification
• Adapt under Attack and Domain Shift: Unified Adversarial Meta-Learning and Domain Adaptation for Robust Automatic Modulation Classification
• Diffusion Models are Robust Pretrainers
• Probabilistic Robustness for Free? Revisiting Training via a Benchmark
• T-MLA: A targeted multiscale log-exponential attack framework for neural image compression
• Secure Distributed RIS-MIMO over Double Scattering Channels: Adversarial Attack, Defense, and SER Improvement
• Parameter Interpolation Adversarial Training for Robust Image Classification
• Lightweight CNN Model Hashing with Higher-Order Statistics and Chaotic Mapping for Piracy Detection and Tamper Localization
• BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing
• ANCHOR: Integrating Adversarial Training with Hard-mined Supervised Contrastive Learning for Robust Representation Learning
• TIGA: Trajectory-Injected Generative Attack against Black-box AIGC Detectors
• Test-Time Defense Against Adversarial Attacks via Stochastic Resonance of Latent Ensembles
• Bilevel Models for Adversarial Learning and A Case Study
• Enhancing CLIP Robustness via Cross-Modality Alignment
• A Versatile Framework for Designing Group-Sparse Adversarial Attacks
• Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
• Toward Understanding the Transferability of Adversarial Suffixes in Large Language Models
• Towards Strong Certified Defense with Universal Asymmetric Randomization
• FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models
• A New Type of Adversarial Examples
• The Black Tuesday Attack: how to crash the stock market with adversarial examples to financial forecasting models
• Nearly Space-Optimal Graph and Hypergraph Sparsification in Insertion-Only Data Streams
• Black-Box Evasion Attacks on Data-Driven Open RAN Apps: Tailored Design and Experimental Evaluation
• Universal and Transferable Attacks on Pathology Foundation Models
• When Flatness Does (Not) Guarantee Adversarial Robustness
• Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Models
• KoALA: KL-L0 Adversarial Detector via Label Agreement
• MS-GAGA: Metric-Selective Guided Adversarial Generation Attack
• StealthAttack: Robust 3D Gaussian Splatting Poisoning via Density-Guided Illusions
• Neutral Agent-based Adversarial Policy Learning against Deep Reinforcement Learning in Multi-party Open Systems
• Adversarial Attacks on Downstream Weather Forecasting Models: Application to Tropical Cyclone Trajectory Prediction
• Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networks
• SynthID-Image: Image watermarking at internet scale
• MemLoss: Enhancing Adversarial Training with Recycling Adversarial Examples
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
• Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
• RegMix: Adversarial Mutual and Generalization Regularization for Enhancing DNN Robustness
• ECLipsE-Gen-Local: Efficient Compositional Local Lipschitz Estimates for Deep Neural Networks
• Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting
• A Statistical Method for Attack-Agnostic Adversarial Attack Detection with Compressive Sensing Comparison
• Sequence-Preserving Dual-FoV Defense for Traffic Sign and Light Recognition in Autonomous Vehicles
• Towards Adversarial Training under Hyperspectral Images
• Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustness
• On the Adversarial Robustness of Learning-based Conformal Novelty Detection
• Cloud Investigation Automation Framework (CIAF): An AI-Driven Approach to Cloud Forensics
• A Call to Action for a Secure-by-Design Generative AI Paradigm
• Robust Federated Inference
• CHAI: Command Hijacking against embodied AI
• SoK: Systematic analysis of adversarial threats against deep learning approaches for autonomous anomaly detection systems in SDN-IoT networks
• Enhancing Certifiable Semantic Robustness via Robust Pruning of Deep Neural Networks
• DeepProv: Behavioral Characterization and Repair of Neural Networks via Inference Provenance Graph Analysis
• VAGUEGAN: Stealthy Poisoning and Backdoor Attacks on Image Generative Pipelines
• AI Safety, Alignment, and Ethics (AI SAE)
• Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
• Visual CoT Makes VLMs Smarter but More Fragile
• Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling
• FERD: Fairness-Enhanced Data-Free Robustness Distillation
• Understanding and Improving Adversarial Robustness of Neural Probabilistic Circuits
• JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
• Adversarial Defense in Cybersecurity: A Systematic Review of GANs for Threat Detection and Mitigation
• Dynamic Dual-level Defense Routing for Continual Adversarial Training
• Localizing Adversarial Attacks To Produces More Imperceptible Noise
• Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
• SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation
• Robustness Feature Adapter for Efficient Adversarial Training
• Is It Certainly a Deepfake? Reliability Analysis in Detection & Generation Ecosystem
• SOLAR: Switchable Output Layer for Accuracy and Robustness in Once-for-All Training
• Explainable Deep Learning Based Adversarial Defense for Automatic Modulation Classification
• Semantic Representation Attack against Aligned Large Language Models
• Adversarial Examples Are Not Bugs, They Are Superposition
• DiffHash: Text-Guided Targeted Attack via Diffusion Models against Deep Hashing Image Retrieval
• AdaGAT: Adaptive Guidance Adversarial Training for the Robustness of Deep Neural Networks
• CIARD: Cyclic Iterative Adversarial Robustness Distillation
• Sharpness-Aware Geometric Defense for Robust Out-Of-Distribution Detection
• A Practical Adversarial Attack against Sequence-based Deep Learning Malware Classifiers
• DARD: Dice Adversarial Robustness Distillation against Adversarial Attacks
• Nearest Neighbor Projection Removal Adversarial Training
• IGAff: Benchmarking Adversarial Iterative and Genetic Affine Algorithms on Deep Neural Networks
• Adversarial Attacks on Audio Deepfake Detection: A Benchmark and Comparative Study
• Realism to Deception: Investigating Deepfake Detectors Against Face Enhancement
• From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
• Measuring the Vulnerability Disclosure Policies of AI Vendors
• Dataset Ownership in the Era of Large Language Models
• MAIA: An Inpainting-Based Approach for Music Adversarial Attacks
• Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack
• On the Learnability of Distribution Classes with Adaptive Adversaries
• ANNIE: Be Careful of Your Robots
• TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space
• A modified exact penalty approach for general constrained ℓ₀-sparse optimization problems
• FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models
• Targeted Physical Evasion Attacks in the Near-Infrared Domain
• An Investigation of Visual Foundation Models Robustness
• SoK: Understanding the Fundamentals and Implications of Sensor Out-of-band Vulnerabilities
• Sequential Difference Maximization: Generating Adversarial Examples via Multi-Stage Optimization
• Learning from Peers: Collaborative Ensemble Adversarial Training
• Assessing the Noise Robustness of Class Activation Maps: A Framework for Reliable Model Interpretability
• Does simple trump complex? Comparing strategies for adversarial robustness in DNNs
• On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking
• TAIGen: Training-Free Adversarial Image Generation via Diffusion Models
• Foe for Fraud: Transferable Adversarial Attacks in Credit Card Fraud Detection
• CIA+TA Risk Assessment for AI Reasoning Vulnerabilities
• DASH: A Meta-Attack Framework for Synthesizing Effective and Stealthy Adversarial Examples
• Semantically Guided Adversarial Testing of Vision Models Using Language Models
• Fre-CW: Targeted Attack on Time Series Forecasting using Frequency Domain Loss
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• Certifiably robust malware detectors by design
• Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification
• DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification
• Slice or the Whole Pie? Utility Control for AI Models
• Guided Perturbation Sensitivity (GPS): Detecting Adversarial Text via Embedding Stability and Word Importance
• When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs
• Adversarial Attacks on Reinforcement Learning-based Medical Questionnaire Systems: Input-level Perturbation Strategies and Medical Constraint Validation
• The Power of Many: Synergistic Unification of Diverse Augmentations for Efficient Adversarial Robustness
• GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbations
• Investigating the Invertibility of Multimodal Latent Spaces: Limitations of Optimization-Based Methods
• Federated Learning on Riemannian Manifolds: A Gradient-Free Projection-Based Approach
• RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function
• Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss
• Teach Me to Trick: Exploring Adversarial Transferability via Knowledge Distillation
• Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation
• NCCR: to Evaluate the Robustness of Neural Networks and Adversarial Examples
• Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation

Related

• Distillation as a Defense to Adversarial Perturbations against Deep\n Neural Networks
• Enhanced Attacks on Defensively Distilled Deep Neural Networks
• Denoising Autoencoder-based Defensive Distillation as an Adversarial Robustness Algorithm
• Increasing Confidence in Adversarial Robustness Evaluations
• Why Blocking Targeted Adversarial Perturbations Impairs the Ability to Learn