Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis

2025/10/01 by Carlo Bosio, Bosio, Carlo, Matteo Guarrera +5
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Systems and Control (eess.SY) #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2510.00373

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language models (LLMs) have shown promise as generators of symbolic control policies, producing interpretable program-like representations through iterative search. However, these models are not capable of separating the functional structure of a policy from the numerical values it is parametrized by, thus making the search process slow and inefficient. We propose a hybrid approach that decouples structural synthesis from parameter optimization by introducing an additional optimization layer for local parameter search. In our method, the numerical parameters of LLM-generated programs are extracted and optimized numerically to maximize task performance. With this integration, an LLM iterates over the functional structure of programs, while a separate optimization loop is used to find a locally optimal set of parameters accompanying candidate programs. We evaluate our method on a set of control tasks, showing that it achieves higher returns and improved sample efficiency compared to purely LLM-guided search. We show that combining symbolic program synthesis with numerical optimization yields interpretable yet high-performing policies, bridging the gap between language-model-guided design and classical control tuning. Our code is available at https://sites.google.com/berkeley.edu/colmo.

Citations

• AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs
• Synthesizing Interpretable Control Policies through Large Language Model Guided Search
• Automatic Gain Tuning for Humanoid Robots Walking Architectures Using Gradient-Free Optimization Techniques
• Qwen2.5-Coder Technical Report
• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery
• StarCoder 2 and The Stack v2: The Next Generation
• Mathematical discoveries from program search with large language models
• StarCoder: may the source be with you!
• Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Language Models Can Teach Themselves to Program Better
• Evolution through Large Models
• InCoder: A Generative Model for Code Infilling and Synthesis
• STaR: Bootstrapping Reasoning With Reasoning
• AI Feynman: a Physics-Inspired Method for Symbolic Regression
• A Tutorial on Bayesian Optimization
• Verifiable Reinforcement Learning via Policy Extraction
• Programmatically Interpretable Reinforcement Learning
• DeepMind Control Suite
• Towards A Rigorous Science of Interpretable Machine Learning
• DeepCoder: Learning to Write Programs
• Bayesian Optimization with Safety Constraints: Safe and Automatic\n Parameter Tuning in Robotics
• Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model

Related

• Synthesizing Interpretable Control Policies through Large Language Model Guided Search
• Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies
• Are Language Models Up to Sequential Optimization Problems? From Evaluation to a Hegelian-Inspired Enhancement
• Linear Feedback Control Systems for Iterative Prompt Optimization in Large Language Models
• CALM: Co-evolution of Algorithms and Language Model for Automatic Heuristic Design