Performance Evaluation of Adversarial Attacks: Discrepancies and Solutions

2021/04/22 by Jing Wu, Wu, Jing, Mingyi Zhou +9
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Cryptography and Security (cs.CR) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2104.11103

openalex publication_date 2021/04/22 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

Recently, adversarial attack methods have been developed to challenge the robustness of machine learning models. However, mainstream evaluation criteria experience limitations, even yielding discrepancies among results under different settings. By examining various attack algorithms, including gradient-based and query-based attacks, we notice the lack of a consensus on a uniform standard for unbiased performance evaluation. Accordingly, we propose a Piece-wise Sampling Curving (PSC) toolkit to effectively address the aforementioned discrepancy, by generating a comprehensive comparison among adversaries in a given range. In addition, the PSC toolkit offers options for balancing the computational cost and evaluation effectiveness. Experimental results demonstrate our PSC toolkit presents comprehensive comparisons of attack algorithms, significantly reducing discrepancies in practice.

Citations

• Adversarial Manipulation of Deep Representations
• Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
• NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
• Sign-OPT: A Query-Efficient Hard-label Adversarial Attack
• Learning Multiple Layers of Features from Tiny Images

Related

• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Attack Strength vs. Detectability Dilemma in Adversarial Machine Learning
• Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
• Fast and Reliable Evaluation of Adversarial Robustness with Minimum-Margin Attack
• Poisoning Attacks on Algorithmic Fairness