Smoothie: Label Free Language Model Routing

2024/12/06 by Neel Guha, Guha, Neel, Mayee F. Chen +7 · 8 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2412.04692

openalex publication_date 2024/12/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.

Cited by

• Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
• LLMBoost: Make Large Language Models Stronger with Boosting
• Uncertainty Quantification for Retrieval-Augmented Reasoning
• Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensembling
• Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback
• Cost-Aware Contrastive Routing for LLMs
• SmartLLMs Scheduler: A Framework for Cost-Effective LLMs Utilization
• MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verification

Related

• Large Language Model Routing with Benchmark Datasets
• CARGO: A Framework for Confidence-Aware Routing of Large Language Models
• Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
• LLMRank: Understanding LLM Strengths for Model Routing
• Lookahead Routing for Large Language Models