CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection

2022/12/09 by Hyekang Kevin Joo, Khoa Vo, Joo, Hyekang Kevin +5 · 15 citations
Computer Science · Engineering · #Anomaly Detection Techniques and Applications #Artificial Immune Systems Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Network Security and Intrusion Detection

paper · pdf · doi:10.48550/arxiv.2212.05136

openalex publication_date 2022/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Video anomaly detection (VAD) -- commonly formulated as a multiple-instance learning problem in a weakly-supervised manner due to its labor-intensive nature -- is a challenging problem in video surveillance where the frames of anomaly need to be localized in an untrimmed video. In this paper, we first propose to utilize the ViT-encoded visual features from CLIP, in contrast with the conventional C3D or I3D features in the domain, to efficiently extract discriminative representations in the novel technique. We then model temporal dependencies and nominate the snippets of interest by leveraging our proposed Temporal Self-Attention (TSA). The ablation study confirms the effectiveness of TSA and ViT feature. The extensive experiments show that our proposed CLIP-TSA outperforms the existing state-of-the-art (SOTA) methods by a large margin on three commonly-used benchmark datasets in the VAD problem (UCF-Crime, ShanghaiTech Campus, and XD-Violence). Our source code is available at https://github.com/joos2010kj/CLIP-TSA.

Cited by

• Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks
• A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis
• TRACES: Temporal Recall with Contextual Embeddings for Real-Time Video Anomaly Detection
• Knowledge-Guided Textual Reasoning for Explainable Video Anomaly Detection via LLMs
• FrameShield: Adversarially Robust Video Anomaly Detection
• MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection
• GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detection
• Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting
• GTA-Crime: A Synthetic Dataset and Generation Framework for Fatal Violence Detection with Adversarial Snippet-Level Domain Adaptation
• Context-Aware Zero-Shot Anomaly Detection in Surveillance Using Contrastive and Predictive Spatiotemporal Modeling
• Aligning Effective Tokens with Video Anomaly in Large Language Models
• Mixture of Experts Guided by Gaussian Splatters Matters: A new Approach to Weakly-Supervised Video Anomaly Detection
• GV-VAD : Exploring Video Generation for Weakly-Supervised Video Anomaly Detection
• The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
• VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding

Related

• Delving into CLIP latent space for Video Anomaly Recognition
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection
• Anomaly detection in surveillance videos using transformer based attention model
• Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning
• Real-world Video Anomaly Detection by Extracting Salient Features in Videos