A systematic assessment of Large Language Models for constructing two-level fractional factorial designs

2025/12/18 by Vazquez, Alan R., Rother, Kilian M., Charles-Gonzalez, Marco V.
#Computation (stat.CO) #FOS: Computer and information sciences #Methodology (stat.ME)

paper · doi:10.48550/arxiv.2512.17113

Abstract

Two-level fractional factorial designs permit the study multiple factors using a limited number of runs. Traditionally, these designs are obtained from catalogs available in standard textbooks or statistical software. However, modern Large Language Models (LLMs) can now produce two-level fractional factorial designs, but the quality of these designs has not been previously assessed. In this paper, we perform a systematic evaluation of two popular classes of LLMs, namely GPT and Gemini models, to construct two-level fractional factorial designs with 8, 16, and 32 runs, and 4 to 26 factors. To this end, we use prompting techniques to develop a high-quality set of design construction tasks for the LLMs. We compare the designs obtained by the LLMs with the best-known designs in terms of resolution and minimum aberration criteria. We show that the LLMs can effectively construct optimal 8-, 16-, and 32-run designs with up to eight factors.

Citations

• A Survey on Large Language Model Benchmarks
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Prompt Engineering Guidelines for Using Large Language Models in Requirements Engineering
• Performance Evaluation of Large Language Models in Statistical Programming
• Robust and Adaptive Optimization under a Large Language Model Lens
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
• RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
• Better Zero-Shot Reasoning with Role-Play Prompting
• LLaMA: Open and Efficient Foundation Language Models
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• Large Language Models are Zero-Shot Reasoners
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Attention Is All You Need

Related

• Exploiting Edited Large Language Models as General Scientific Optimizers
• Specializing Smaller Language Models towards Multi-Step Reasoning
• Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
• How Can Large Language Models Help Humans in Design and Manufacturing?
• LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model