EvoPrompting: Language Models for Code-Level Neural Architecture Search

2023/02/28 by Angelica Chen, D. Dohan, David M. Dohan +4 · 1 voice · 31 citations
Computer Science · Engineering · #Adaptation (eye) #Architecture #Artificial intelligence #Artificial neural network #Benchmark (surveying) #Code (set theory) #Computer science #Convolutional neural network #Crossover #Evolutionary Algorithms and Applications #Evolutionary algorithm #Ferroelectric and Negative Capacitance Devices #Language model #MNIST database #Machine learning #Metaheuristic Optimization Algorithms Research #Programming language #Task (project management) #Variety (cybernetics) #cs.AI #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2302.14838

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/02/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Given the recent impressive accomplishments of language models (LMs) for code generation, we explore the use of LMs as adaptive mutation and crossover operators for an evolutionary neural architecture search (NAS) algorithm. While NAS still proves too difficult a task for LMs to succeed at solely through prompting, we find that the combination of evolutionary prompt engineering with soft prompt-tuning, a method we term EvoPrompting, consistently finds diverse and high performing models. We first demonstrate that EvoPrompting is effective on the computationally efficient MNIST-1D dataset, where EvoPrompting produces convolutional architecture variants that outperform both those designed by human experts and naive few-shot prompting in terms of accuracy and model size. We then apply our method to searching for graph neural networks on the CLRS Algorithmic Reasoning Benchmark, where EvoPrompting is able to design novel architectures that outperform current state-of-the-art models on 21 out of 30 algorithmic reasoning tasks while maintaining similar model size. EvoPrompting is successful at designing accurate and efficient neural network architectures across a variety of machine learning tasks, while also being general enough for easy adaptation to other tasks beyond neural network design.

Cited by

• Autonomous Discovery of Wireless Communications Algorithms
• Scaling Closed-Loop Feature Channel Configuration with LLMs
• A Toolbox for Improving Evolutionary Prompt Search
• Informing Acquisition Functions via Foundation Models for Molecular Discovery
• CogMCTS: A Novel Cognitive-Guided Monte Carlo Tree Search Framework for Iterative Heuristic Evolution with Large Language Models
• RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design
• optimize_anything: Unified Text Optimization can Outperform Specialized Systems
• FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data
• Hypothesis Hunting with Evolving Networks of Autonomous Scientific Agents
• ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
• On Discovering Algorithms for Adversarial Imitation Learning
• CoLLM-NAS: Collaborative Large Language Models for Efficient Knowledge-Guided Neural Architecture Search
• Structuring Collective Action with LLM-Guided Evolution: From Ill-Structured Problems to Executable Heuristics
• Optimizing ground state preparation protocols with autoresearch
• Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
• Large Language Model Assisted Automated Algorithm Generation and Evolution via Meta-black-box optimization
• Agent²: An Agent-Generates-Agent Framework for Reinforcement Learning Automation
• T-CACE: A Time-Conditioned Autoregressive Contrast Enhancement Multi-Task Framework for Contrast-Free Liver MRI Synthesis, Segmentation, and Diagnosis
• \(X\)-evolve: Solution space evolution powered by large language models
• DHEvo: Data-Algorithm Based Heuristic Evolution for Generalizable MILP Solving
• PhaseNAS: Language-Model Driven Architecture Search with Dynamic Phase Adaptation
• AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks
• AlphaEvolve: A coding agent for scientific and algorithmic discovery
• iDSE: Navigating Design Space Exploration in High-Level Synthesis Using LLMs
• InnateCoder: Learning Programmatic Options with Foundation Models
• Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch
• Language Model Crossover: Variation through Few-Shot Prompting
• Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
• ABG-NAS: Adaptive Bayesian Genetic Neural Architecture Search for Graph Representation Learning
• Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery
• Transferrable Surrogates in Expressive Neural Architecture Search Spaces

Discussions

• EvoPrompting: Language Models for Code-Level Neural Architecture Search [hn, 21 points, 1 comments]

Related

• EvoMerge: Neuroevolution for Large Language Models
• Evolutionary Computation and Large Language Models: A Survey of Methods, Synergies, and Applications
• Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap
• RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design
• LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection