From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning

2025/10/02 by Rafael Rodríguez‐Sánchez, Rodriguez-Sanchez, Rafael, Cameron Allen +3 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2510.02484

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.

Citations

• SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions
• Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning
• NAVIX: Scaling MiniGrid Environments with JAX
• Learning Abstract World Model for Value-preserving Planning with Options
• Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning
• Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies
• General Identifiability and Achievability for Causal Representation Learning
• ELDEN: Exploration via Local Dependencies
• Identifiability Guarantees for Causal Disentanglement from Soft Interventions
• Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
• BISCUIT: Causal Representation Learning from Binary Interactions
• Granger Causal Interaction Skill Chains
• Learning World Models with Identifiable Factorization
• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing
• Nonparametric Identifiability of Causal Representations from Unknown Interventions
• Disentanglement via Latent Quantization
• Conditional Mutual Information for Disentangled Representations in Reinforcement Learning
• Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning
• Linear Causal Disentanglement via Interventions
• MoCoDA: Model-based Counterfactual Data Augmentation
• Interventional Causal Representation Learning
• Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning
• Denoised MDPs: Learning World Models Better Than the World Itself
• Causal Dynamics Learning for Task-Independent State Abstraction
• Variational Causal Dynamics: Discovering Modular World Models from Interventions
• Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems
• Temporal Difference Learning for Model Predictive Control
• CITRIS: Causal Identifiability from Temporal Intervened Sequences
• Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Toward Causal Representation Learning
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Towards Minimax Optimal Reinforcement Learning in Factored Markov Decision Processes
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• Reinforcement Learning in Factored MDPs: Oracle-Efficient Algorithms and Tighter Regret Bounds for the Non-Episodic Setting
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Learning Latent Dynamics for Planning from Pixels
• Representation Learning with Contrastive Predictive Coding
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• Disentangling the independently controllable factors of variation by interacting with the world
• Disentangling by Factorising
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Gaussian Error Linear Units (GELUs)
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Continuous control with deep reinforcement learning
• End-to-End Training of Deep Visuomotor Policies
• Near-optimal Reinforcement Learning in Factored MDPs
• Representation Learning: A Review and New Perspectives
• Representation Learning: A Review and New Perspectives
• Policy Iteration for Factored MDPs
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

Cited by

• From monoliths to modules: Decomposing transducers for efficient world modelling

Related

• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• Independently Controllable Factors
• Learning World Models with Identifiable Factorization
• Learning State Representations from Random Deep Action-conditional Predictions
• TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning