BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generation

2025/10/09 by Rocktim Jyoti Das, Das, Rocktim Jyoti, Harsh Singh +13
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2510.08572

openalex publication_date 2025/10/09 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Scaling data and models has played a pivotal role in the remarkable progress of computer vision and language. Inspired by these domains, recent efforts in robotics have similarly focused on scaling both data and model size to develop more generalizable and robust policies. However, unlike vision and language, robotics lacks access to internet-scale demonstrations across diverse robotic tasks and environments. As a result, the scale of existing datasets typically suffers from the need for manual data collection and curation. To address this problem, here we propose BLAZER, a framework that learns manipulation policies from automatically generated training data. We build on the zero-shot capabilities of LLM planners and automatically generate demonstrations for diverse manipulation tasks in simulation. Successful examples are then used to finetune an LLM and to improve its planning capabilities without human supervision. Notably, while BLAZER training requires access to the simulator's state, we demonstrate direct transfer of acquired skills to sensor-based manipulation. Through extensive experiments, we show BLAZER to significantly improve zero-shot manipulation in both simulated and real environments. Moreover, BLAZER improves on tasks outside of its training pool and enables downscaling of LLM models. Our code and data will be made publicly available on the project page.

Citations

• MolmoAct: Action Reasoning Models that can Reason in Space
• A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation
• ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• MALT: Improving Reasoning with Multi-Agent LLM Training
• MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation
• Why Does the Effective Context Length of LLMs Fall Short?
• GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
• AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data
• Lean-STaR: Learning to Interleave Thinking and Proving
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
• RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
• OpenVLA: An Open-Source Vision-Language-Action Model
• A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation
• Towards Generalist Robot Learning from Internet Video: A Survey
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
• RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
• M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• Physically Grounded Vision-Language Models for Robotic Manipulation
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
• REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Micrograph segmentations for DDEVD
• Segment Anything
• GPT-4 Technical Report
• Large Language Models Are Reasoning Teachers
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
• Code as Policies: Language Model Programs for Embodied Control
• Instruction-driven history-aware policies for robotic manipulations
• STaR: Bootstrapping Reasoning With Reasoning
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Implicit Behavioral Cloning
• RLBench: The Robot Learning Benchmark & Learning Environment
• Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700\n Robot Hours
• End-to-End Training of Deep Visuomotor Policies

Related

• LLM Trainer: Automated Robotic Data Generating via Demonstration Augmentation using LLMs
• Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
• Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
• GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
• Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models