2018/06/01 by Christopher Stanton, Jeff Clune, Stanton, Christopher +2 · 3 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Digital Games and Media #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1806.00553
openalex publication_date 2018/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Traditional exploration methods in RL require agents to perform random\nactions to find rewards. But these approaches struggle on sparse-reward domains\nlike Montezuma's Revenge where the probability that any random action sequence\nleads to reward is extremely low. Recent algorithms have performed well on such\ntasks by encouraging agents to visit new states or perform new actions in\nrelation to all prior training episodes (which we call across-training\nnovelty). But such algorithms do not consider whether an agent exhibits\nintra-life novelty: doing something new within the current episode, regardless\nof whether those behaviors have been performed in previous episodes. We\nhypothesize that across-training novelty might discourage agents from\nrevisiting initially non-rewarding states that could become important stepping\nstones later in training. We introduce Deep Curiosity Search (DeepCS), which\nencourages intra-life exploration by rewarding agents for visiting as many\ndifferent states as possible within each episode, and show that DeepCS matches\nthe performance of current state-of-the-art methods on Montezuma's Revenge. We\nfurther show that DeepCS improves exploration on Amidar, Freeway, Gravitar, and\nTutankham (many of which are hard exploration games). Surprisingly, DeepCS\ndoubles A2C performance on Seaquest, a game we would not have expected to\nbenefit from intra-life exploration because the arena is small and already\neasily navigated by naive exploration techniques. In one run, DeepCS achieves a\nmaximum training score of 80,000 points on Seaquest, higher than any methods\nother than Ape-X. The strong performance of DeepCS on these sparse- and\ndense-reward tasks suggests that encouraging intra-life novelty is an\ninteresting, new approach for improving performance in Deep RL and motivates\nfurther research into hybridizing across-training and intra-life exploration\nmethods.\n