2021/06/01 by Van-Quang Nguyen, Masanori Suganuma, Nguyen, Van-Quang +3
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2106.00596
openalex publication_date 2021/06/01 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
There is a growing interest in the community in making an embodied AI agent\nperform a complicated task while interacting with an environment following\nnatural language directives. Recent studies have tackled the problem using\nALFRED, a well-designed dataset for the task, but achieved only very low\naccuracy. This paper proposes a new method, which outperforms the previous\nmethods by a large margin. It is based on a combination of several new ideas.\nOne is a two-stage interpretation of the provided instructions. The method\nfirst selects and interprets an instruction without using visual information,\nyielding a tentative action sequence prediction. It then integrates the\nprediction with the visual information etc., yielding the final prediction of\nan action and an object. As the object's class to interact is identified in the\nfirst stage, it can accurately select the correct object from the input image.\nMoreover, our method considers multiple egocentric views of the environment and\nextracts essential information by applying hierarchical attention conditioned\non the current instruction. This contributes to the accurate prediction of\nactions for navigation. A preliminary version of the method won the ALFRED\nChallenge 2020. The current version achieves the unseen environment's success\nrate of 4.45% with a single view, which is further improved to 8.37% with\nmultiple views.\n