Mental Models of Adversarial Machine Learning

2021/05/08 by Lukas Bieringer, Kathrin Grosse, Bieringer, Lukas +5
Computer Science · Social Sciences · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Information and Cyber Security

paper · pdf · doi:10.48550/arxiv.2105.03726

openalex publication_date 2021/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root causes or improve risk communication. Our study reveals two \facets of practitioners' mental models of machine learning security. Firstly, practitioners often confuse machine learning security with threats and defences that are not directly related to machine learning. Secondly, in contrast to most academic research, our participants perceive security of machine learning as not solely related to individual models, but rather in the context of entire workflows that consist of multiple components. Jointly with our additional findings, these two facets provide a foundation to substantiate mental models for machine learning security and have implications for the integration of adversarial machine learning into corporate workflows, \newdecreasing practitioners' reported uncertainty, and appropriate regulatory frameworks for machine learning security.

Citations

• A Coefficient of Agreement for Nominal Scales
• The Measurement of Observer Agreement for Categorical Data
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Motivating the Rules of the Game for Adversarial Example Research
• Bad Global Minima Exist and SGD Can Reach Them
• Towards Reverse-Engineering Black-Box Neural Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Adversarial Machine Learning -- Industry Perspectives
• What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research

Related

• Adversarial Machine Learning -- Industry Perspectives
• Security and Machine Learning in the Real World
• Towards the Science of Security and Privacy in Machine Learning
• Risk Management Framework for Machine Learning Security
• Security Matters: A Survey on Adversarial Machine Learning