Why Build an Assistant in Minecraft?

2019/07/22 by Arthur Szlam, Szlam, Arthur, Jonathan Gray +25 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Computation and Language (cs.CL) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms

paper · pdf · doi:10.48550/arxiv.1907.09273

openalex publication_date 2019/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural language understanding and learning from dialogue.

Citations

• Learning to Map Sentences to Logical Form: Structured Classification\n with Probabilistic Categorial Grammars
• Playing Atari with Deep Reinforcement Learning
• MazeBase: A Sandbox for Learning from Games
• Human-level concept learning through probabilistic program induction
• Neural Programmer-Interpreters
• Language to Logical Form with Neural Attention
• Exploring the Limits of Language Modeling
• Data Recombination for Neural Semantic Parsing
• TerpreT: A Probabilistic Programming Language for Program Induction
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• Differentiable Programs with Neural Libraries
• Teaching Machines to Describe Images via Natural Language Feedback
• StarCraft II: A New Challenge for Reinforcement Learning
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• HoME: a Household Multimodal Environment
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoders
• Towards Deep Conversational Recommendations
• End-to-end optimization of goal-driven and visually grounded dialogue systems Harm de Vries
• Learning Dependency-Based Compositional Semantics
• TextWorld: A Learning Environment for Text-based Games
• Interactive Grounded Language Acquisition and Generalization in a 2D World

Cited by

• Talking-to-Build: How LLM-Assisted Interface Shapes Player Performance and Experience in Minecraft
• MDC-R: The Minecraft Dialogue Corpus with Reference

Related

• CraftAssist: A Framework for Dialogue-enabled Interactive Agents
• Odyssey: Empowering Minecraft Agents with Open-World Skills
• STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
• CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant
• Improving Agent Interactions in Virtual Environments with Language Models