LMPVC and Policy Bank: Adaptive voice control for industrial robots with code generating LLMs and reusable Pythonic policies

2025/06/27 by Ossi Parikka, Parikka, Ossi, Roel Pieters +1
Computer Science · #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Robotics (cs.RO) #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2506.22028

openalex publication_date 2025/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern industry is increasingly moving away from mass manufacturing, towards more specialized and personalized products. As manufacturing tasks become more complex, full automation is not always an option, human involvement may be required. This has increased the need for advanced human robot collaboration (HRC), and with it, improved methods for interaction, such as voice control. Recent advances in natural language processing, driven by artificial intelligence (AI), have the potential to answer this demand. Large language models (LLMs) have rapidly developed very impressive general reasoning capabilities, and many methods of applying this to robotics have been proposed, including through the use of code generation. This paper presents Language Model Program Voice Control (LMPVC), an LLM-based prototype voice control architecture with integrated policy programming and teaching capabilities, built for use with Robot Operating System 2 (ROS2) compatible robots. The architecture builds on prior works using code generation for voice control by implementing an additional programming and teaching system, the Policy Bank. We find this system can compensate for the limitations of the underlying LLM, and allow LMPVC to adapt to different downstream tasks without a slow and costly training process. The architecture and additional results are released on GitHub (https://github.com/ozzyuni/LMPVC).

Citations

• SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
• Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
• RVT-2: Learning Precise Manipulation from Few Demonstrations
• StarCoder 2 and The Stack v2: The Next Generation
• RobotGPT: Robot Manipulation Learning from ChatGPT
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Structured Chain-of-Thought Prompting for Code Generation
• Structured Chain-of-Thought Prompting for Code Generation
• PaLM-E: An Embodied Multimodal Language Model
• ChatGPT for Robotics: Design Principles and Model Abilities
• Robust Speech Recognition via Large-Scale Weak Supervision
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
• Code as Policies: Language Model Programs for Embodied Control
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
• CLIPort: What and Where Pathways for Robotic Manipulation
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Related

• Interpreting and learning voice commands with a Large Language Model for a robot system
• Code as Policies: Language Model Programs for Embodied Control
• GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Models
• Reducing Latency in LLM-Based Natural Language Commands Processing for Robot Navigation
• RDMM: Fine-Tuned LLM Models for On-Device Robotic Decision Making with Enhanced Contextual Awareness in Specific Domains