Predicting Video Slot Attention Queries from Random Slot-Feature Pairs

2025/08/02 by Rongzhen Zhao, Zhao, Rongzhen, Jian Li +5 · 3 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Feature learning #Frame (networking) #Human Pose and Action Recognition #Multimodal Machine Learning Applications #News aggregator #Object (grammar) #Query optimization #Representation (politics) #Semantics (computer science) #Video tracking

paper · pdf · doi:10.48550/arxiv.2508.01345

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/08/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Unsupervised video Object-Centric Learning (OCL) is promising as it enables object-level scene representation and understanding as we humans do. Mainstream video OCL methods adopt a recurrent architecture: An aggregator aggregates current video frame into object features, termed slots, under some queries; A transitioner transits current slots to queries for the next frame. This is an effective architecture but all existing implementations both (i1) neglect to incorporate next frame features, the most informative source for query prediction, and (i2) fail to learn transition dynamics, the knowledge essential for query prediction. To address these issues, we propose Random Slot-Feature pair for learning Query prediction (RandSF.Q): (t1) We design a new transitioner to incorporate both slots and features, which provides more information for query prediction; (t2) We train the transitioner to predict queries from slot-feature pairs randomly sampled from available recurrences, which drives it to learn transition dynamics. Experiments on scene representation demonstrate that our method surpass existing video OCL methods significantly, e.g., up to 10 points on object discovery, setting new state-of-the-art. Such superiority also benefits downstream tasks like scene understanding. Source Code, Model Checkpoints, Training Logs: https://github.com/Genera1Z/RandSF.Q

Citations

• Slot Attention with Re-Initialization and Self-Distillation
• SlotPi: Physics-informed Object-centric Reasoning Models
• Temporally Consistent Object-Centric Learning by Contrasting Slots
• Grouped Discrete Representation for Object-Centric Learning
• Multi-Scale Fusion for Object Representation
• Reasoning-Enhanced Object-Centric Learning for Videos
• SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers
• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos
• Multi-Object Discovery by Low-Dimensional Object Motion
• Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities
• SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models
• SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models
• Bridging the Gap to Real-World Object-Centric Learning
• SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos
• Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
• Learning What and Where: Disentangling Location and Identity Tracking Without Supervision
• Unsupervised Learning of Accurate Siamese Tracking
• Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning
• Conditional Object-Centric Learning from Video
• Illiterate DALL-E Learns to Compose
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective
• Attention over learned object embeddings enables complex visual reasoning
• Unsupervised Deep Representation Learning for Real-Time Tracking
• Object-Centric Learning with Slot Attention
• MAST: A Memory-Augmented Self-supervised Tracker
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• SCALOR: Generative World Models with Scalable Object Representations
• Joint-task Self-supervised Learning for Temporal Correspondence
• MONet: Unsupervised Scene Decomposition and Representation
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Attention Is All You Need

Cited by

• TSA: Temporal Slot Activation for Persistent Object-Centric Video Representation
• Object-Centric Vision Token Pruning for Vision Language Models
• Smoothing Slot Attention Iterations and Recurrences

Related

• Smoothing Slot Attention Iterations and Recurrences
• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos
• Slot Attention with Re-Initialization and Self-Distillation
• Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learning
• Improving Object-centric Learning with Query Optimization