2020/10/02 by Vincent Cartillier, Cartillier, Vincent, Zhile Ren +9 · 2 citations
Engineering · Computer Science · #Robotics and Sensor-Based Localization #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications
paper · pdf · doi:10.48550/arxiv.2010.01191
We study the task of semantic mapping - specifically, an embodied agent (a\nrobot or an egocentric AI assistant) is given a tour of a new environment and\nasked to build an allocentric top-down semantic map ("what is where?") from\negocentric observations of an RGB-D camera with known pose (via localization\nsensors). Towards this goal, we present SemanticMapNet (SMNet), which consists\nof: (1) an Egocentric Visual Encoder that encodes each egocentric RGB-D frame,\n(2) a Feature Projector that projects egocentric features to appropriate\nlocations on a floor-plan, (3) a Spatial Memory Tensor of size floor-plan\nlength x width x feature-dims that learns to accumulate projected egocentric\nfeatures, and (4) a Map Decoder that uses the memory tensor to produce semantic\ntop-down maps. SMNet combines the strengths of (known) projective camera\ngeometry and neural representation learning. On the task of semantic mapping in\nthe Matterport3D dataset, SMNet significantly outperforms competitive baselines\nby 4.01-16.81% (absolute) on mean-IoU and 3.81-19.69% (absolute) on Boundary-F1\nmetrics. Moreover, we show how to use the neural episodic memories and\nspatio-semantic allocentric representations build by SMNet for subsequent tasks\nin the same space - navigating to objects seen during the tour("Find chair") or\nanswering questions about the space ("How many chairs did you see in the\nhouse?"). Project page: https://vincentcartillier.github.io/smnet.html.\n