Using Large Language Models for Hyperparameter Optimization

2023/12/07 by Michael R. Zhang, Nishkrit Desai, Zhang, Michael R. +7 · 13 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.04528

openalex publication_date 2023/12/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

Cited by

• Encrypted Traffic Detection in Resource Constrained IoT Networks: A Diffusion Model and LLM Integrated Framework
• Informing Acquisition Functions via Foundation Models for Molecular Discovery
• LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
• Automated Discovery of Test Oracles for Database Management Systems Using LLMs
• GRPOformer: Advancing Hyperparameter Optimization via Group Relative Policy Optimization
• Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning
• Large Language Models in Operations Research: Methods, Applications, and Challenges
• Agent²: An Agent-Generates-Agent Framework for Reinforcement Learning Automation
• MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimization
• Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search
• Towards Execution-Grounded Automated AI Research
• Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
• Large Language Model Empowered Design of Fluid Antenna Systems: Challenges, Frameworks, and Case Studies for 6G

Related

• Crafting Efficient Fine-Tuning Strategies for Large Language Models
• Sequential Large Language Model-Based Hyper-parameter Optimization
• An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models
• Large Language Model Agent for Hyper-Parameter Optimization
• Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications