Choosing the Sample Size of a Computer Experiment: A Practical Guide

2009/11/01 by Jason L. Loeppky, Jerome Sacks, William J. Welch · 760 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Multi-Objective Optimization Algorithms #Algorithm #Code (set theory) #Computer experiment #Computer science #Dimension (graph theory) #Gaussian #Gaussian Processes and Bayesian Inference #Key (lock) #Machine learning #Mathematics #Optimal Experimental Design Methods #Process (computing) #Programming language #Sample (material) #Sample size determination #Sensitivity (control systems) #Simulation #Source code #Statistics #Theoretical computer science #Variable (mathematics) #Variables

paper · doi:10.1198/tech.2009.08040

published in Technometrics 51(4), 366-376 (Taylor & Francis)

openalex publication_date 2009/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

We provide reasons and evidence supporting the informal rule that the number of runs for an effective initial computer experiment should be about 10 times the input dimension. Our arguments quantify two key characteristics of computer codes that affect the sample size required for a desired level of accuracy when approximating the code via a Gaussian process (GP). The first characteristic is the total sensitivity of a code output variable to all input variables; the second corresponds to the way this total sensitivity is distributed across the input variables, specifically the possible presence of a few prominent input factors and many impotent ones (i.e., effect sparsity). Both measures relate directly to the correlation structure in the GP approximation of the code. In this way, the article moves toward a more formal treatment of sample size for a computer experiment. The evidence supporting these arguments stems primarily from a simulation study and via specific codes modeling climate and ligand activation of G-protein.

Cited by

• Two-Stage Sensitivity-Based Group Screening in Computer Experiments
• Grouped Latin Hypercube Designs with Controlled Correlations
• BKP: An R Package for Beta Kernel Process Modeling
• Using Machine Learning to Generate a GISS ModelE Calibrated Physics Ensemble (CPE)
• Functional Gaussian Process Model for Bayesian Nonparametric Analysis
• A hierarchical expected improvement method for Bayesian optimization
• Mind the Jumps: A Scalable Robust Local Gaussian Process for Multidimensional Response Surfaces with Discontinuities
• Initial conditions of bulk matter in ultrarelativistic nuclear collisions
• Physics-informed Polynomial Chaos Expansion with Enhanced Constrained Optimization Solver and D-optimal Sampling
• A Review on Quantile Regression for Stochastic Computer Experiments
• A class of space-filling designs and their projection properties
• Comparison of surrogate-based uncertainty quantification methods for computationally expensive simulators
• Gaussian Process Subspace Prediction for Model Reduction
• Regional heterogeneity in left atrial stiffness impacts passive deformation in a cohort of patient-specific models
• Rapid calibration of atrial electrophysiology models using Gaussian process emulators in the ensemble Kalman filter
• Modeling with uncertainty quantification reveals the essentials of a non-canonical algal carbon-concentrating mechanism
• Sequential design of computer experiments for the estimation of a probability of failure
• Efficient estimation of divergence-based sensitivity indices with Gaussian process surrogates
• Global Sensitivity Analysis of Stochastic Computer Models with joint metamodels
• Data-Driven Analysis and Common Proper Orthogonal Decomposition (CPOD)-Based Spatio-Temporal Emulator for Design Exploration
• MAGI-X: Manifold-Constrained Gaussian Process Inference for Unknown System Dynamics
• Unsupervised Urban Tree Biodiversity Mapping from Street-Level Imagery Using Spatially-Aware Visual Clustering
• Efficient optimization of expensive black-box simulators via marginal means, with application to neutrino detector design
• Statistical Uncertainty Analysis for Stochastic Simulation
• Diagnostics for Stochastic Gaussian Process Emulators
• Advanced methodology for uncertainty propagation in computer experiments with large number of inputs
• Robust designs for Gaussian process emulation of computer experiments
• Quantifying the Quark Gluon Plasma
• The Problem with Assessing Statistical Methods
• Bayesian Active Learning of (small) Quantile Sets through Expected Estimator Modification
• Monotone Function Estimation for Computer Experiments
• The ICSCREAM methodology: Identification of penalizing configurations in computer experiments using screening and metamodel -- Applications in thermal-hydraulics
• Kernel-smoothed proper orthogonal decomposition (KSPOD)-based emulation for prediction of spatiotemporally evolving flow dynamics
• Improved Surrogate Modeling using Machine Learning for Industrial Civil Aircraft Aerodynamics
• Bayesian Subset Simulation: a kriging-based subset simulation algorithm for the estimation of small probabilities of failure
• History Matching of A Complex Epidemiological Model of Human Immunodeficiency Virus Transmission By Using Variance Emulation
• Efficient Optimization of the Likelihood Function in Gaussian Process Modelling
• Bayesian Optimization in Linear Time
• Common kernel-smoothed proper orthogonal decomposition (CKSPOD): An efficient reduced-order model for emulation of spatiotemporally evolving flow dynamics
• Bayesian Subset Simulation
• GPfit: AnRPackage for Fitting a Gaussian Process Model to Deterministic Simulator Outputs
• Learning Explainable Dense Reward Shapes via Bayesian Optimization
• Computer model calibration based on image warping metrics: an application for sea ice deformation
• Statistical emulation of landslide-induced tsunamis at the Rockall Bank, NE Atlantic. [europepmc]
• Using compartmental models to simulate directed acyclic graphs to explore competing causal mechanisms underlying epidemiological study data. [europepmc]
• Tropospheric ozone in CCMI models and Gaussian process emulation to understand biases in the SOCOLv3 chemistry-climate model. [europepmc]
• Exploratory-Phase-Free Estimation of GP Hyperparameters in Sequential Design Methods-At the Example of Bayesian Inverse Problems. [europepmc]
• Statistical Emulation of Winter Ambient Fine Particulate Matter Concentrations From Emission Changes in China. [europepmc]
• Maximum expected entropy transformed Latin hypercube designs. [europepmc]
• Sensitivity of Air Pollution Exposure and Disease Burden to Emission Changes in China Using Machine Learning Emulation. [europepmc]
• Projecting Climate Dependent Coastal Flood Risk With a Hybrid Statistical Dynamical Model. [europepmc]
• Air volume flow rate optimization of the guide vanes in an axial flow fan based on DOE and CFD. [europepmc]
• Modeling with uncertainty quantification reveals the essentials of a non-canonical algal carbon-concentrating mechanism. [europepmc]
• Regional heterogeneity in left atrial stiffness impacts passive deformation in a cohort of patient-specific models. [europepmc]
• Rapid calibration of atrial electrophysiology models using Gaussian process emulators in the ensemble Kalman filter. [europepmc]
• Advancing Sensitivity Analysis of an Intervertebral Disc Finite Element Model: A Comparative Approach Using Neural Networks [europepmc]

Related

• A Framework for Controlling Sources of Inaccuracy in Gaussian Process Emulation of Deterministic Computer Experiments
• Factorial Sampling Plans for Preliminary Computational Experiments
• Statistical sensitiveness for science
• Cases for the nugget in modeling computer experiments
• Power, precision, and sample size estimation in sport and exercise science research