Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack

2025/09/28 by Yukun Chen, Boheng Li, Chen, Yukun +13 · 1 citation
Computer Science · Engineering · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Smart Grid Security and Resilience

paper · pdf · doi:10.48550/arxiv.2509.23871

openalex publication_date 2025/09/28 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Knowledge distillation (KD) is a vital technique for deploying deep neural networks (DNNs) on resource-constrained devices by transferring knowledge from large teacher models to lightweight student models. While teacher models from third-party platforms may undergo security verification (\eg, backdoor detection), we uncover a novel and critical threat: distillation-conditional backdoor attacks (DCBAs). DCBA injects dormant and undetectable backdoors into teacher models, which become activated in student models via the KD process, even with clean distillation datasets. While the direct extension of existing methods is ineffective for DCBA, we implement this attack by formulating it as a bilevel optimization problem and proposing a simple yet effective method (\ie, SCAR). Specifically, the inner optimization simulates the KD process by optimizing a surrogate student model, while the outer optimization leverages outputs from this surrogate to optimize the teacher model for implanting the conditional backdoor. Our SCAR addresses this complex optimization utilizing an implicit differentiation algorithm with a pre-optimized trigger injection function. Extensive experiments across diverse datasets, model architectures, and KD techniques validate the effectiveness of our SCAR and its resistance against existing backdoor detection, highlighting a significant yet previously overlooked vulnerability in the KD process. Our code is available at https://github.com/WhitolfChen/SCAR.

Citations

• Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
• A Comprehensive Survey on Knowledge Distillation
• REFINE: Inversion-Free Backdoor Defense via Model Reprogramming
• FLARE: Toward Universal Dataset Purification against Backdoor Attacks
• Transferring Backdoors between Large Language Models by Knowledge Distillation
• BAN: Detecting Backdoors Activated by Adversarial Neuron Noise
• Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks
• IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistency
• TrustAgent: Towards Safe and Trustworthy LLM-based Agents
• Robust Backdoor Detection for Deep Learning via Topological Evolution Dynamics
• Towards Sample-specific Backdoor Attack with Clean Labels via Attribute Trigger
• Watch Out! Simple Horizontal Class Backdoor Can Trivially Evade Defense
• Deep Learning Safety Concerns in Automated Driving Perception
• MDTD: A Multi Domain Trojan Detector for Deep Neural Networks
• Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound
• Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability
• Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
• SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency
• BackdoorBox: A Python Toolbox for Backdoor Learning
• Mind Your Heart: Stealthy Backdoor Attack on Dynamic Deep Neural Network in Edge Computing
• BATT: Backdoor Attack with Transformation-based Triggers
• CASSOCK: Viable Backdoor Attacks against DNN in The Wall of Source-Specific Backdoor Defences
• BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learning
• Knowledge Distillation from A Stronger Teacher
• Knowledge Distillation with the Reused Teacher Classifier
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
• Adversarial Unlearning of Backdoors via Implicit Hypergradient
• Quantization Backdoors to Deep Learning Commercial Frameworks
• Stealthy Backdoors as Compression Artifacts
• A Backdoor Attack against 3D Point Cloud Classifiers
• Backdoor Attack in the Physical World
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Backdoor Learning: A Survey
• On the Iteration Complexity of Hypergradient Computation
• Knowledge Distillation: A Survey
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Detecting AI Trojans Using Meta Neural Analysis
• Similarity-Preserving Knowledge Distillation
• Relational Knowledge Distillation
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Hyperparameter optimization with approximate gradient
• Deep Residual Learning for Image Recognition
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by

• BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning

Related

• BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning
• How to Backdoor the Knowledge Distillation
• INK: Inheritable Natural Backdoor Attack Against Model Distillation
• Transferring Backdoors between Large Language Models by Knowledge Distillation
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks