Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

2018/11/08 by Bryant Chen, Wilka Carvalho, Chen, Bryant +14 · 231 citations
Computer Science · Mathematics · #Advanced Malware Detection Techniques #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Adversary #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Backdoor #Cluster analysis #Computer science #Computer security #Data mining #Inference #Machine learning #Perspective (graphical) #Set (abstract data type) #Training set #cs.CR #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1811.03728

published in arXiv (Cornell University) (Cornell University)

arxiv created 2018/11/09 · openalex publication_date 2018/11/09 · arxiv updated 2018/11/12 · openalex created_date 2022/08/02 · openalex updated_date 2026/08/06

Abstract

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

Cited by

• Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation
• ArcGen: Generalizing Neural Backdoor Detection Across Diverse Architectures
• Quantization Blindspots: How Model Compression Breaks Backdoor Defenses
• Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models
• Dynamic Black-box Backdoor Attacks on IoT Sensory Data
• Breaking the Stealth-Potency Trade-off in Clean-Image Backdoors with Generative Trigger Optimization
• Signature in Code Backdoor Detection, how far are we?
• Injection, Attack and Erasure: Revocable Backdoor Attacks via Machine Unlearning
• TabVLA: Targeted Backdoor Attacks on Vision-Language-Action Models
• Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models
• GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis
• Provable Watermarking for Data Poisoning Attacks
• Geometry-Aware Backdoor Attacks: Leveraging Curvature in Hyperbolic Embeddings
• Backdoor Attacks Against Speech Language Models
• Temporal Poisoning: Clean-Label Backdoors via Event Redistribution in SNNs
• Adversarial machine learning :
• Backdoor Attacks and Defenses in Computer Vision Domain: A Survey
• Prototype-Guided Robust Learning against Backdoor Attacks
• Silent Until Sparse: Backdoor Attacks on Semi-Structured Sparsity
• Detecting Stealthy Data Poisoning Attacks in AI Code Generators
• RepoMark: A Data-Usage Auditing Framework for Code Large Language Models
• Backdooring Self-Supervised Contrastive Learning by Noisy Alignment
• From Detection to Correction: Backdoor-Resilient Face Recognition via Vision-Language Trigger Detection and Noise-Based Neutralization
• NT-ML: Backdoor Defense via Non-target Label Training and Mutual Learning
• BDFirewall: Towards Effective and Expeditiously Black-Box Backdoor Defense in MLaaS
• Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defense
• NCCR: to Evaluate the Robustness of Neural Networks and Adversarial Examples
• Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense
• Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment
• MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing
• 3S-Attack: Spatial, Spectral and Semantic Invisible Backdoor Attack Against DNN Models
• Backdoor Scanning for Deep Neural Networks through K-Arm Optimization
• Mask6D: Masked Pose Priors For 6D Object Pose Estimation
• When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
• Identifying Physically Realizable Triggers for Backdoored Face Recognition Networks
• Circumventing Backdoor Space via Weight Symmetry
• CUBA: Controlled Untargeted Backdoor Attack against Deep Neural Networks
• TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models
• TED-LaST: Towards Robust Backdoor Defense Against Adaptive Attacks
• Robust Watermarks Meet Backdoored Models: Evading Diffusion Semantic Watermarks via Stealthy Backdoor
• A Chaos Driven Metric for Backdoor Attack Detection
• Backdoors in Code Summarizers: How Bad Is It?
• Variance-Based Defense Against Blended Backdoor Attacks
• A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluation Methods
• Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies
• Poison in the Well: Feature Embedding Disruption in Backdoor Attacks
• FL-PLAS: Federated Learning with Partial Layer Aggregation for Backdoor Defense Against High-Ratio Malicious Clients
• On the Security Risks of ML-based Malware Detection Systems: A Survey
• Cyber Threat Intelligence for Artificial Intelligence Systems
• Lightweight and Fast Backdoor Model Detection
• FFCBA: Feature-based Full-target Clean-label Backdoor Attacks
• SFIBA: Spatial-based Full-target Invisible Backdoor Attacks
• Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
• Secure Transfer Learning: Training Clean Models Against Backdoor in (Both) Pre-trained Encoders and Downstream Datasets

Related

• SGBA: A Stealthy Scapegoat Backdoor Attack against Deep Neural Networks
• Detection of Backdoors in Trained Classifiers Without Access to the Training Set
• Invisible Backdoor Attacks Using Data Poisoning in the Frequency Domain
• Backdoor Learning: A Survey
• Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation