VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

2026/07/16 by Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4
#cs.CV #cs.AI

paper · pdf

Abstract

We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard 224² to 1024² on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.

Citations

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• On the Construction and Comparison of Difference Schemes
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• VideoMamba: Spatio-Temporal Selective State Space Model
• VideoMamba: State Space Model for Efficient Video Understanding
• SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
• VMoBA: Mixture-of-Block Attention for Video Diffusion Models
• Look Every Frame All at Once: Video-Ma²mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing