Choosing the Sample Size of a Computer Experiment: A Practical Guide

2009/11/01 by Jason L. Loeppky, Jerome Sacks, William J. Welch · 26 citations
Computer Science · Decision Sciences · #Advanced Multi-Objective Optimization Algorithms #Gaussian Processes and Bayesian Inference #Optimal Experimental Design Methods

paper · doi:10.1198/tech.2009.08040

Abstract

We provide reasons and evidence supporting the informal rule that the number of runs for an effective initial computer experiment should be about 10 times the input dimension. Our arguments quantify two key characteristics of computer codes that affect the sample size required for a desired level of accuracy when approximating the code via a Gaussian process (GP). The first characteristic is the total sensitivity of a code output variable to all input variables; the second corresponds to the way this total sensitivity is distributed across the input variables, specifically the possible presence of a few prominent input factors and many impotent ones (i.e., effect sparsity). Both measures relate directly to the correlation structure in the GP approximation of the code. In this way, the article moves toward a more formal treatment of sample size for a computer experiment. The evidence supporting these arguments stems primarily from a simulation study and via specific codes modeling climate and ligand activation of G-protein.

Cited by

• Two-Stage Sensitivity-Based Group Screening in Computer Experiments
• Grouped Latin Hypercube Designs with Controlled Correlations
• BKP: An R Package for Beta Kernel Process Modeling
• Using Machine Learning to Generate a GISS ModelE Calibrated Physics Ensemble (CPE)
• Functional Gaussian Process Model for Bayesian Nonparametric Analysis
• A hierarchical expected improvement method for Bayesian optimization
• Mind the Jumps: A Scalable Robust Local Gaussian Process for Multidimensional Response Surfaces with Discontinuities
• Initial conditions of bulk matter in ultrarelativistic nuclear collisions
• Physics-informed Polynomial Chaos Expansion with Enhanced Constrained Optimization Solver and D-optimal Sampling
• A Review on Quantile Regression for Stochastic Computer Experiments
• A class of space-filling designs and their projection properties
• Comparison of surrogate-based uncertainty quantification methods for computationally expensive simulators
• Gaussian Process Subspace Prediction for Model Reduction
• Regional heterogeneity in left atrial stiffness impacts passive deformation in a cohort of patient-specific models
• Rapid calibration of atrial electrophysiology models using Gaussian process emulators in the ensemble Kalman filter
• Modeling with uncertainty quantification reveals the essentials of a non-canonical algal carbon-concentrating mechanism
• Sequential design of computer experiments for the estimation of a probability of failure
• Efficient estimation of divergence-based sensitivity indices with Gaussian process surrogates
• Global Sensitivity Analysis of Stochastic Computer Models with joint metamodels
• Data-Driven Analysis and Common Proper Orthogonal Decomposition (CPOD)-Based Spatio-Temporal Emulator for Design Exploration
• MAGI-X: Manifold-Constrained Gaussian Process Inference for Unknown System Dynamics
• Unsupervised Urban Tree Biodiversity Mapping from Street-Level Imagery Using Spatially-Aware Visual Clustering
• Efficient optimization of expensive black-box simulators via marginal means, with application to neutrino detector design
• Statistical Uncertainty Analysis for Stochastic Simulation
• Diagnostics for Stochastic Gaussian Process Emulators
• Advanced methodology for uncertainty propagation in computer experiments with large number of inputs
• Robust designs for Gaussian process emulation of computer experiments

Related

• A Framework for Controlling Sources of Inaccuracy in Gaussian Process Emulation of Deterministic Computer Experiments
• Factorial Sampling Plans for Preliminary Computational Experiments
• Statistical sensitiveness for science
• Cases for the nugget in modeling computer experiments
• Power, precision, and sample size estimation in sport and exercise science research