Hierarchical Reinforcement Learning for Air Combat at DARPA's AlphaDogfight Trials

2021/05/03 by Adrian P. Pope, Jaime S. Ide, Pope, Adrian P. +20 · 1 citation
Engineering · #Guidance and Control Systems #Military Defense Systems Analysis #Aerospace and Aviation Technology

paper · pdf · doi:10.1109/tai.2022.3222143

Abstract

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event.

Cited by

• Learning-Based Multi-Stage Strategy for a Fixed-Wing Aircraft to Evade a Missile Detected at a Short Distance

Related

• Enhancing Aerial Combat Tactics through Hierarchical Multi-Agent Reinforcement Learning
• A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat
• A deep reinforcement learning control approach for high-performance aircraft
• Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering
• Reinforcement Learning based Air Combat Maneuver Generation