On the security relevance of weights in deep learning

2019/02/08 by Kathrin Grosse, Thomas Alexander Trost, Grosse, Kathrin +7
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.1902.03020

openalex publication_date 2019/02/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, a weight-based attack on stochastic gradient descent inducing overfitting has been proposed. We show that the threat is broader: A task-independent permutation on the initial weights suffices to limit the achieved accuracy to for example 50% on the Fashion MNIST dataset from initially more than 90%. These findings are confirmed on MNIST and CIFAR. We formally confirm that the attack succeeds with high likelihood and does not depend on the data. Empirically, weight statistics and loss appear unsuspicious, making it hard to detect the attack if the user is not aware. Our paper is thus a call for action to acknowledge the importance of the initial weights in deep learning.

Citations

• On the difficulty of training Recurrent Neural Networks
• Poisoning Attacks against Support Vector Machines
• Gradient-based learning applied to document recognition
• A Study of Deep Learning Robustness Against Computation Failures
• Bad Global Minima Exist and SGD Can Reach Them
• Learning Multiple Layers of Features from Tiny Images

Related

• Weight Poisoning Attacks on Pre-trained Models
• "Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice
• Wolf in Sheep's Clothing - The Downscaling Attack Against Deep Learning Applications
• On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping
• Gradient Similarity: An Explainable Approach to Detect Adversarial\n Attacks against Deep Learning