LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation

2024/03/02 by Zeyuan Ma, Ma, Zeyuan, Hongshu Guo +11 · 21 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model-Driven Software Engineering Techniques #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC) #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2403.01131

openalex publication_date 2024/03/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent research explores optimization using large language models (LLMs) by either iteratively seeking next-step solutions from LLMs or directly prompting LLMs for an optimizer. However, these approaches exhibit inherent limitations, including low operational efficiency, high sensitivity to prompt design, and a lack of domain-specific knowledge. We introduce LLaMoCo, the first instruction-tuning framework designed to adapt LLMs for solving optimization problems in a code-to-code manner. Specifically, we establish a comprehensive instruction set containing well-described problem prompts and effective optimization codes. We then develop a novel two-phase learning strategy that incorporates a contrastive learning-based warm-up procedure before the instruction-tuning phase to enhance the convergence behavior during model fine-tuning. The experiment results demonstrate that a CodeGen (350M) model fine-tuned by our LLaMoCo achieves superior optimization performance compared to GPT-4 Turbo and the other competitors across both synthetic and realistic problem sets. The fine-tuned model and the usage instructions are available at https://anonymous.4open.science/r/LLaMoCo-722A.

Cited by

• Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Design
• PerfCoder: Large Language Models for Interpretable Code Performance Optimization
• Code Refactoring with LLM: A Comprehensive Evaluation With Few-Shot Settings
• AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solving
• SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search
• An Agentic Framework with LLMs for Solving Complex Vehicle Routing Problems
• A Stochastic Differential Equation Framework for Multi-Objective LLM Interactions: Dynamical Systems Analysis with Code Generation Applications
• Opal: A Modular Framework for Optimizing Performance using Analytics and LLMs
• AP2O-Coder: Adaptively Progressive Preference Optimization for Reducing Compilation and Runtime Errors in LLM-Generated Code
• OptGraph: Large Language Models Enhanced Evolutionary Optimization Via Graph Retrieval-Augmented Generation
• Instance Generation for Meta-Black-Box Optimization through Latent Space Reverse Engineering
• Large Language Models in Operations Research: Methods, Applications, and Challenges
• Language-Guided Tuning: Enhancing Numeric Optimization with Textual Feedback
• OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modeling
• ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle
• Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
• DesignX: Human-Competitive Algorithm Designer for Black-Box Optimization
• MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization
• Efficient Heuristics Generation for Solving Combinatorial Optimization Problems Using Large Language Models
• RideAgent: An LLM-Enhanced Optimization Framework for Automated Taxi Fleet Operations
• NED-Tree: Bridging the Semantic Gap with Nonlinear Element Decomposition Tree for LLM Nonlinear Optimization Modeling

Related

• Large Language Models for Compiler Optimization
• LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch
• Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers
• A Problem-Oriented Perspective and Anchor Verification for Code Optimization
• PerfCoder: Large Language Models for Interpretable Code Performance Optimization