LLM Chemistry Estimation for Multi-LLM Recommendation

2025/10/04 by Huascar Sanchez, Briland Hitaj, Sanchez, Huascar +1 · 1 citation
Chemistry · Computer Science · #Analytical Chemistry and Chromatography #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #Web Data Mining and Analysis

paper · pdf · doi:10.48550/arxiv.2510.03930

openalex publication_date 2025/10/04 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Multi-LLM collaboration promises accurate, robust, and context-aware solutions, yet existing approaches rely on implicit selection and output assessment without analyzing whether collaborating models truly complement or conflict. We introduce LLM Chemistry -- a framework that measures when LLM combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities. We formalize the notion of chemistry among LLMs, propose algorithms that quantify it by analyzing interaction dependencies, and recommend optimal model ensembles accordingly. Our theoretical analysis shows that chemistry among collaborating LLMs is most evident under heterogeneous model profiles, with its outcome impact shaped by task type, group size, and complexity. Evaluation on classification, summarization, and program repair tasks provides initial evidence for these task-dependent effects, thereby reinforcing our theoretical results. This establishes LLM Chemistry as both a diagnostic factor in multi-LLM systems and a foundation for ensemble recommendation.

Citations

• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
• Orchestration for Domain-specific Edge-Cloud Language Models
• Harnessing Multiple Large Language Models: A Survey on LLM Ensemble
• Aligning Compound AI Systems via System-level DPO
• Towards Resource-Efficient Compound AI Systems
• Multi-Agent Collaboration Mechanisms: A Survey of LLMs
• A Unified Approach to Routing and Cascading for LLMs
• Quantifying Generalization Complexity for Large Language Models
• Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models
• RouteLLM: Learning to Route LLMs with Preference Data
• Improving Multi-Agent Debate with Sparse Communication Topology
• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
• Learning Multi-Agent Communication from Graph Modeling Perspective
• A Survey on Efficient Inference for Large Language Models
• Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization
• RouterBench: A Benchmark for Multi-LLM Routing System
• Purifying Large Language Models by Ensembling a Small Language Model
• Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration
• Model Merging by Uncertainty-Based Gradient Matching
• Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration
• Explainability for Large Language Models: A Survey
• AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
• Self-Refine: Iterative Refinement with Self-Feedback
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News\n Detection
• Diversity and dissimilarity coefficients: A unified approach

Cited by

• Multi-LLM Collaboration for Medication Recommendation

Related

• When One LLM Drools, Multi-LLM Collaboration Rules
• Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm
• Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
• A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesis