Episodic Exploration for Deep Deterministic Policies: An Application to\n StarCraft Micromanagement Tasks

2016/09/09 by Nicolas Usunier, Usunier, Nicolas, Gabriel Synnaeve +5 · 1 voice · 16 citations
Computer Science · Economics, Econometrics and Finance · Social Sciences · #Artificial Intelligence in Games #Sports Analytics and Performance #Digital Games and Media

paper · pdf · doi:10.48550/arxiv.1609.02993

Abstract

We consider scenarios from the real-time strategy game StarCraft as new\nbenchmarks for reinforcement learning algorithms. We propose micromanagement\ntasks, which present the problem of the short-term, low-level control of army\nmembers during a battle. From a reinforcement learning point of view, these\nscenarios are challenging because the state-action space is very large, and\nbecause there is no obvious feature representation for the state-action\nevaluation function. We describe our approach to tackle the micromanagement\nscenarios with deep neural network controllers from raw state features given by\nthe game engine. In addition, we present a heuristic reinforcement learning\nalgorithm which combines direct exploration in the policy space and\nbackpropagation. This algorithm allows for the collection of traces for\nlearning using deterministic policies, which appears much more efficient than,\nfor example, \ε-greedy exploration. Experiments show that with this\nalgorithm, we successfully learn non-trivial strategies for scenarios with\narmies of up to 15 agents, where both Q-learning and REINFORCE struggle.\n

Citations

• Stochastic Estimation of the Maximum of a Regression Function

Cited by

• A Survey of Deep Reinforcement Learning in Video Games
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Value-Decomposition Multi-Agent Actor-Critics
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• STARDATA: A StarCraft AI Research Dataset
• Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications
• Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games
• Defining Digital Quadruplets in the Cyber-Physical-Social Space for Parallel Driving
• Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• Combining Strategic Learning and Tactical Search in Real-Time Strategy Games

Discussions

• Deep Reinforcement Learning to Play StarCraft [hn, 116 points, 50 comments]

Related

• Learning Macromanagement in StarCraft from Replays using Deep Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Learning to Guide Multiple Heterogeneous Actors from a Single Human Demonstration via Automatic Curriculum Learning in StarCraft II