RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization

2025/10/03 by Fukazawa, Kai, Kunal Mundada, Mundada, Kunal +2
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.02695

openalex publication_date 2025/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) is attractive only if policies achieve high returns without catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of either (i) value/model-based pessimism or (ii) restricted policy classes that limit expressiveness, whereas diffusion/flow-based expressive generative policies have largely been used in risk-neutral settings. We introduce Risk-Aware Multimodal Actor-Critic (RAMAC), a simple, modular, model-free framework that couples an expressive generative actor (e.g., diffusion/flow) with a distributional critic and optimizes a composite objective that combines Conditional Value-at-Risk (CVaR) with behavioral cloning (BC), enabling risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further provide an objective-level analysis showing that controlling behavior divergence via BC suppresses OOD actions and stabilizes CVaR. Instantiating RAMAC with a diffusion actor, we illustrate these insights on a 2-D risky bandit and evaluate on Stochastic-D4RL, observing consistent gains in CVaR_0.1 while maintaining strong returns. The code and experimental results are available on the \hrefhttps://kaifukazawa.github.io/ramac-project/ project website

Citations

• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training
• Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers
• VITA: Vision-to-Action Flow Matching Policy
• Distributional Soft Actor-Critic with Diffusion Policy
• Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
• Flow Q-Learning
• Efficient Online Reinforcement Learning for Diffusion Policy
• FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulation
• InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
• Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
• JUICER: Data-Efficient Imitation Learning for Robotic Assembly
• Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
• A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization
• Efficient Diffusion Policies for Offline Reinforcement Learning
• Revisiting the Minimalist Approach to Offline Reinforcement Learning
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
• Is Conditional Generative Modeling all you need for Decision-Making?
• Flow Matching for Generative Modeling
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Planning with Diffusion for Flexible Behavior Synthesis
• RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
• Conservative Offline Distributional Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Diffusion Models Beat GANs on Image Synthesis
• COMBO: Conservative Offline Model-Based Policy Optimization
• Risk-Averse Offline Reinforcement Learning
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Conservative Q-Learning for Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Statistics and Samples in Distributional Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Safe Policy Improvement with Baseline Bootstrapping
• A Distributional Perspective on Reinforcement Learning
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Policy Gradient for Coherent Risk Measures
• Optimizing the CVaR via Sampling
• Policy Gradients with Variance Related Risk Criteria

Related

• Risk-Averse Offline Reinforcement Learning
• One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
• Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
• Conservative Offline Distributional Reinforcement Learning
• Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning