2023/01/27 by Dongseok Shim, Seung‐Jae Lee, Shim, Dongseok +3 · 2 citations
Computer Science · Physics and Astronomy · #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2301.11520
openalex publication_date 2023/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.