SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples

2025/07/10 by Dren Fazlija, Monty-Maximilian Zühlke, Fazlija, Dren +11
Computer Science · Social Sciences · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2507.07776

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

SCOOTER Evaluation and Dataset Creation Process Data This data collection contains the following information associated with our six initial SCOOTER studies: The real image baseline (ImageNet S-R50-N), containing 2966 ImageNet instances that we used to generate adversarial examples (AEs) Notes on how we processed the validation set images to generate ImageNet S-R50-N (see notes.txt and removed_or_modified.txt) All AEs generated within our six SCOOTER studies (see the directories semanticadv/, cadv/, ncf/, diffattack/, advpp/, and aca/) + the collected ratings per image (see the Salman2020Do_R50_annotations.csv file per directory) The ratings generated by GPT-4o (see 4o_ratings/) The predictions of our victim model for all ImageNet Validation set images that displayed exactly one ImageNet object (>39,000 images, see predictions_salman_clean.csv) The first four files are all contained in images_v1.0.0.zip, while the last file is stored separately. More information on how to process the information at hand is available at our GitHub repository: https://github.com/DrenFazlija/SCOOTER SCOOTER Data 1.0.1 The repository now includes an additional file called scooter_db_entries.zip, which contains SQL table data that can be directly used to set up the web app: modified.csv: All 6,924 generated modified images with their respective metadata (e.g., associated attack name and victim model) real.csv: All 2,966 ImageNet S-R50-N images with their respective metadata. atc.csv: All instruction manipulation check and bogus item images. ishihara.csv: All Ishihara-like images. cc.csv: All final comprehension check images. For more information about the web app, check out the dedicated section of our repository: https://github.com/DrenFazlija/SCOOTER/tree/main/ui SCOOTER Data 1.0.2 The repository now includes the previously missing annotation data for our AdvPP experiment (see advpp_annotations.csv). Please use the file to replace the empty Salman2020Do_R50_annotations.csv file in the advpp/ directory. SCOOTER Data 1.1.0 The repository now includes annotation data from a new, transfer-based attack called GA-DMI-FSA (refer to Appendix G.1.2 of the latest pre-print version for details): ga_dmi_fsa.zip contains all 106 successful AEs, while anonymized_ga-dmi-fsa_with_image_ids.csv contains the corresponding, anonymized SCOOTER annotations. Citation Please cite the original paper when using (parts of) our data: @miscfazlija2026scooterhumanevaluationframework, title=SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples, author=Dren Fazlija and Monty-Maximilian Zühlke and Johanna Schrader and Arkadij Orlov and Clara Stein and Iyiola E. Olatunji and Daniel Kudenko, year=2026, eprint=2507.07776, archivePrefix=arXiv, primaryClass=cs.CV, url=https://arxiv.org/abs/2507.07776,

Citations

• How Real Is Real? A Human Evaluation Framework for Unrestricted Adversarial Examples
• An Analysis of Human Alignment of Latent Diffusion Models
• DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification
• TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment
• TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment
• AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Models
• Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability
• Content-based Unrestricted Adversarial Attack
• Diffusion Models for Imperceptible and Transferable Adversarial Attack
• Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation
• Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA
• Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks
• Unrestricted Black-box Adversarial Attack Using GAN with Limited Queries
• Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices
• Towards Transferable Unrestricted Adversarial Examples with Minimum Changes
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Unrestricted Adversarial Attacks on ImageNet Competition
• Training data-efficient image transformers & distillation through attention
• MTurk Research: Review and Recommendations
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• RobustBench: a standardized adversarial robustness benchmark
• SoK: Certified Robustness for Deep Neural Networks
• Do Adversarially Robust ImageNet Models Transfer Better?
• Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
• Denoising Diffusion Probabilistic Models
• Are we done with ImageNet?
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
• Reliable Fidelity and Diversity Metrics for Generative Models
• ColorFool: Semantic Adversarial Colorization
• Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distance
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Semantic Adversarial Attacks: Parametric Transformations That Fool Deep\n Classifiers
• Improved Precision and Recall Metric for Assessing Generative Models
• Unrestricted Adversarial Examples via Semantic Manipulation
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Wasserstein Adversarial Examples via Projected Sinkhorn Iterations
• Do ImageNet Classifiers Generalize to ImageNet?
• Attacks on State-of-the-Art Face Recognition using Attentional Adversarial Attack Generative Network
• SparseFool: a few pixels make a big difference
• Unrestricted Adversarial Examples
• Assessing Generative Models via Precision and Recall
• Constructing Unrestricted Adversarial Examples with Generative Models
• Adversarial Attacks and Defences Competition
• Semantic Adversarial Examples
• Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
• Demystifying MMD GANs
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Methods for the detection of carelessly invalid responses in survey data
• Towards Evaluating the Robustness of Neural Networks
• A study of the effect of JPG compression on adversarial images
• Deep Residual Learning for Image Recognition
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Explaining and Harnessing Adversarial Examples
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• Fitting Linear Mixed-Effects Models using lme4
• Intriguing properties of neural networks
• A NEW MEASURE OF RANK CORRELATION
• Constructing Semantics-Aware Adversarial Examples with a Probabilistic Perspective
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• How Real Is Real? A Human Evaluation Framework for Unrestricted Adversarial Examples
• Advances in adversarial attacks and defenses in computer vision: A\n survey
• Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
• A Survey of Black-Box Adversarial Attacks on Computer Vision Models
• A Survey of Adversarial Defenses in Vision-based Systems: Categorization, Methods and Challenges