Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows

2025/05/07 by Wenhao Li, Bo Jin, Li, Wenhao +7
Computer Science · Decision Sciences · Materials Science · #Advanced Multi-Objective Optimization Algorithms #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning in Materials Science #Optimization and Control (math.OC) #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2505.04354

openalex publication_date 2025/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This position paper argues that optimization problem solving can transition from expert-dependent to evolutionary agentic workflows. Traditional optimization practices rely on human specialists for problem formulation, algorithm selection, and hyperparameter tuning, creating bottlenecks that impede industrial adoption of cutting-edge methods. We contend that an evolutionary agentic workflow, powered by foundation models and evolutionary search, can autonomously navigate the optimization space, comprising problem, formulation, algorithm, and hyperparameter spaces. Through case studies in cloud resource scheduling and ADMM parameter adaptation, we demonstrate how this approach can bridge the gap between academic innovation and industrial implementation. Our position challenges the status quo of human-centric optimization workflows and advocates for a more scalable, adaptive approach to solving real-world optimization problems.

Citations

• Exploring the Necessity of Reasoning in LLM-based Agent Scenarios
• From System 1 to System 2: A Survey of Reasoning Large Language Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
• Lifelong Learning of Large Language Model based Agents: A Roadmap
• Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey
• Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization
• Sequential Large Language Model-Based Hyper-parameter Optimization
• A Survey of Small Language Models
• SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning
• A Systematic Survey on Large Language Models for Algorithm Design
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
• Multi-objective Evolution of Heuristic Using Large Language Model
• LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
• Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models
• A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
• Towards Lifelong Learning of Large Language Models: A Survey
• Position: A Call to Action for a Human-Centered AutoML Paradigm
• To Believe or Not to Believe Your LLM
• A Survey on Large Language Models for Code Generation
• Exploiting LLM Quantization
• Position: Foundation Agents as the Paradigm Shift for Decision Making
• When Large Language Model Meets Optimization
• Leveraging Large Language Model to Generate a Novel Metaheuristic Algorithm with CRISPE Framework
• Can large language models reason and plan?
• LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems
• Data Interpreter: An LLM Agent For Data Science
• A Survey on Knowledge Distillation of Large Language Models
• OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models
• Large Language Model Agent for Hyper-Parameter Optimization
• ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution
• Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap
• Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects
• Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process
• Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model
• Mathematical discoveries from program search with large language models
• Using Large Language Models for Hyperparameter Optimization
• Algorithm Evolution Using Large Language Model
• Synthesizing mixed-integer linear programming models from natural language descriptions
• Eureka: Human-Level Reward Design via Coding Large Language Models
• When can transformers reason with abstract symbols?
• FireAct: Toward Language Agent Fine-tuning
• OptiMUS: Optimization Modeling Using MIP Solvers and large language models
• Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future
• The Rise and Potential of Large Language Model Based Agents: A Survey
• Large Language Models as Optimizers
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning
• Code Llama: Open Foundation Models for Code
• Diagnosing Infeasible Optimization Problems Using Large Language Models
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
• Holy Grail 2.0: From Natural Language to Constraint Models
• Learning to Model the World with Language
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Reasoning with Language Model is Planning with World Model
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Towards an Automatic Optimisation Model Generator Assisted with Generative Pre-trained Transformer
• AutoML-GPT: Automatic Machine Learning with GPT
• OpenAGI: When LLM Meets Domain Experts
• Generative Agents: Interactive Simulacra of Human Behavior
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions
• Language Model Crossover: Variation through Few-Shot Prompting
• Language Model Crossover: Variation through Few-Shot Prompting
• A Novel Approach for Auto-Formulation of Optimization Problems
• Toolformer: Language Models Can Teach Themselves to Use Tools
• The Evolutionary Computation Methods No One Should Use
• Affective Coherence Monitoring for Transformer-Based Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks
• Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions
• A Survey of ADMM Variants for Distributed Optimization: Problems, Algorithms and Features
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Learning to Optimize: A Primer and A Benchmark
• The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot\n Learners
• Benchmarking in Optimization: Best Practice and Open Issues
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Meta-Learning in Neural Networks: A Survey
• Distributed Learning in the Nonconvex World: From batch data to streaming and beyond
• Optuna: A Next-generation Hyperparameter Optimization Framework
• A Survey of Optimization Methods from a Machine Learning Perspective
• Understanding and correcting pathologies in the training of learned\n optimizers
• Why We Do Not Evolve Software? Analysis of Evolutionary Algorithms
• The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Efficient Neural Architecture Search via Parameter Sharing
• Regularized Evolution for Image Classifier Architecture Search
• Neural Program Meta-Induction
• ADMM Penalty Parameter Selection by Residual Balancing
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to Optimize Neural Nets
• An overview of gradient descent optimization algorithms
• Learning to learn by gradient descent by gradient descent
• Adaptive ADMM with Spectral Penalty Parameter Selection
• A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data
• A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing
• The Configurable SAT Solver Challenge (CSSC)
• Practical Bayesian Optimization of Machine Learning Algorithms
• Выпуклая оптимизация
• Lean Copilot: Large Language Models as Copilots for Theorem Proving in Lean
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• A Simplex Method for Function Minimization

Related

• "It Was a Magical Box": Understanding Practitioner Workflows and Needs in Optimization
• A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving
• OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents
• Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms
• Socio-cognitive agent-oriented evolutionary algorithm with trust-based optimization