LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

2025/10/30 by Youssef Attia El Hili, Albert Thomas, Hili, Youssef Attia El +11
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2510.26510

openalex publication_date 2025/10/30 · openalex created_date 2025/11/01 · openalex updated_date 2026/07/28

Abstract

Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.

Citations

• Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning
• Sequential Large Language Model-Based Hyper-parameter Optimization
• Eliminating Position Bias of Language Models: A Mechanistic Approach
• Large Language Model Agent for Hyper-Parameter Optimization
• A Meta-Level Learning Algorithm for Sequential Hyper-Parameter Space Reduction in AutoML
• Using Large Language Models for Hyperparameter Optimization
• A systematic study comparing hyperparameter optimization engines on tabular data
• Primacy Effect of ChatGPT
• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning
• Meta-in-context learning in large language models
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
• Can GPT-4 Perform Neural Architecture Search?
• General-Purpose In-Context Learning by Meta-Learning Transformers
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation
• Meta-Learning with Adaptive Hyperparameters
• Large Dimensional Analysis and Improvement of Multi Task Learning
• Language Models are Few-Shot Learners
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Taking the Human Out of the Loop: A Review of Bayesian Optimization

Related

• Using Large Language Models for Hyperparameter Optimization
• MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning
• Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
• Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models