Robustness Evaluation for Video Models with Reinforcement Learning

2025/06/05 by Ashwin Ramesh Babu, Babu, Ashwin Ramesh, Sajad Mousavi +13
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2506.05431

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Evaluating the robustness of Video classification models is very challenging, specifically when compared to image-based models. With their increased temporal dimension, there is a significant increase in complexity and computational cost. One of the key challenges is to keep the perturbations to a minimum to induce misclassification. In this work, we propose a multi-agent reinforcement learning approach (spatial and temporal) that cooperatively learns to identify the given video's sensitive spatial and temporal regions. The agents consider temporal coherence in generating fine perturbations, leading to a more effective and visually imperceptible attack. Our method outperforms the state-of-the-art solutions on the Lp metric and the average queries. Our method enables custom distortion types, making the robustness evaluation more relevant to the use case. We extensively evaluate 4 popular models for video action recognition on two popular datasets, HMDB-51 and UCF-101.

Citations

• Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness
• Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos
• Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics
• Boosting the Transferability of Video Adversarial Examples via Temporal Translation
• Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations
• Towards Transferable Adversarial Attacks on Vision Transformers
• Just One Moment: Structural Vulnerability of Deep Action Recognition against One Frame Attack
• Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior
• Sparse Black-box Video Attack with Reinforcement Learning
• Appending Adversarial Frames for Universal Video Attack
• Heuristic Black-box Adversarial Attacks on Video Recognition Models
• Black-box Adversarial Attacks on Video Recognition Models
• SlowFast Networks for Video Recognition
• TSM: Temporal Shift Module for Efficient Video Understanding
• Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors
• Sparse Adversarial Perturbations for Videos
• Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Proximal Policy Optimization Algorithms
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Related

• Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos
• Large-scale Robustness Analysis of Video Action Recognition Models
• Sparse Black-box Video Attack with Reinforcement Learning
• Multi-Agent Reinforcement Learning Based Frame Sampling for Effective\n Untrimmed Video Recognition
• Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed\n Videos