Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation

2019/07/12 by Wenjie Shang, Shang, Wenjie, Yang Yu +9 · 3 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Data Stream Mining Techniques #Discriminator #FOS: Computer and information sciences #Learning environment #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Reinforcement Learning in Robotics #Reinforcement learning #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1907.06584

published in arXiv (Cornell University) (Cornell University) · Appears in KDD 2019

arxiv created 2019/07/12 · openalex publication_date 2019/07/12 · arxiv updated 2019/07/16 · openalex created_date 2019/07/23 · openalex updated_date 2026/08/08

Abstract

Reinforcement learning aims at searching the best policy model for decision making, and has been shown powerful for sequential recommendations. The training of the policy by reinforcement learning, however, is placed in an environment. In many real-world applications, however, the policy training in the real environment can cause an unbearable cost, due to the exploration in the environment. Environment reconstruction from the past data is thus an appealing way to release the power of reinforcement learning in these applications. The reconstruction of the environment is, basically, to extract the casual effect model from the data. However, real-world applications are often too complex to offer fully observable environment information. Therefore, quite possibly there are unobserved confounding variables lying behind the data. The hidden confounder can obstruct an effective reconstruction of the environment. In this paper, by treating the hidden confounder as a hidden policy, we propose a deconfounded multi-agent environment reconstruction (DEMER) approach in order to learn the environment together with the hidden confounder. DEMER adopts a multi-agent generative adversarial imitation learning framework. It proposes to introduce the confounder embedded policy, and use the compatible discriminator for training the policies. We then apply DEMER in an application of driver program recommendation. We firstly use an artificial driver program recommendation environment, abstracted from the real application, to verify and analyze the effectiveness of DEMER. We then test DEMER in the real application of Didi Chuxing. Experiment results show that DEMER can effectively reconstruct the hidden confounder, and thus can build the environment better. DEMER also derives a recommendation policy with a significantly improved performance in the test phase of the real application.

Citations

• Trust Region Policy Optimization
• A survey of robot learning from demonstration
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Causal Effect Inference with Deep Latent-Variable Models
• Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Generative Adversarial Nets

Related

• Plug-and-Play Model-Agnostic Counterfactual Policy Synthesis for Deep Reinforcement Learning based Recommendation
• Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design
• Toward Simulating Environments in Reinforcement Learning Based Recommendations
• Adversarial Environment Design via Regret-Guided Diffusion Models
• PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representations