Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions

2025/06/09 by Yunsock Lee, Lee, Yu-Ang, Grace Y. Yi +7 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.08234

openalex publication_date 2025/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in large language models (LLMs) and AI systems have led to a paradigm shift in the design and optimization of complex AI workflows. By integrating multiple components, compound AI systems have become increasingly adept at performing sophisticated tasks. However, as these systems grow in complexity, new challenges arise in optimizing not only individual components but also their interactions. While traditional optimization methods such as supervised fine-tuning (SFT) and reinforcement learning (RL) remain foundational, the rise of natural language feedback introduces promising new approaches, especially for optimizing non-differentiable systems. This paper provides a systematic review of recent progress in optimizing compound AI systems, encompassing both numerical and language-based techniques. We formalize the notion of compound AI system optimization, classify existing methods along several key dimensions, and highlight open research challenges and future directions in this rapidly evolving field. A list of surveyed papers is publicly available at https://github.com/MiuLab/AISysOpt-Survey.

Citations

• MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming
• FlowReasoner: Reinforcing Query-Level Meta-Agents
• Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
• VFlow: Discovering Optimal Agentic Workflows for Verilog Generation
• Can Textual Gradient Work in Federated Learning?
• MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
• Aligning Compound AI Systems via System-level DPO
• Optimizing Model Selection for Compound AI Systems
• SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning
• Multi-agent Architecture Search via Agentic Supernet
• ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
• Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
• Towards Resource-Efficient Compound AI Systems
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
• Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
• GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
• REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
• How to Correctly do Semantic Backpropagation on Language-based Agentic Systems
• SoK: A Systems Perspective on Compound AI Threats and Countermeasures
• AIME: AI System Optimization via Multiple LLM Evaluators
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
• SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning
• Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
• AutoFlow: Automated Workflow Generation for Large Language Model Agents
• Symbolic Learning Enables Self-Evolving Agents
• Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
• TorchOpera: A Compound AI System for LLM Safety
• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• GAIA: a benchmark for General AI Assistants
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• Joint Prompt Optimization of Stacked LLMs using Variational Inference
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Mindstorms in Natural Language-Based Societies of Mind
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• GPT-4 Technical Report
• Large Language Models Are Human-Level Prompt Engineers
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Measuring Massive Multitask Language Understanding
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Attention Is All You Need
• Constrained Policy Optimization
• Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Survey
• Towards AI-45^∘ Law: A Roadmap to Trustworthy AGI

Cited by

• From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems
• Compass: Optimizing Compound AI Workflows for Dynamic Adaptation
• PLAIground: SLO-Driven Runtime Model Selection for Compound AI Systems in the Edge-Cloud-Space Continuum

Related

• Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards
• When Large Language Model Meets Optimization
• REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
• Optimizing Model Selection for Compound AI Systems
• OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents