Using simulation studies to evaluate statistical methods
2019/01/16 by Tim P. Morris, Ian R. White, Michael J. Crowther · 1 voice · 72 citations
Decision Sciences · Mathematics · #Meta-analysis and systematic reviews #Statistical Methods and Bayesian Inference #Statistical Methods in Clinical Trials
paper · pdf · doi:10.1002/sim.8086
openalex created_date 2017/12/22 · openalex publication_date 2019/01/16 · openalex updated_date 2026/07/31
Abstract
Simulation studies are computer experiments that involve creating data by pseudo-random sampling. A key strength of simulation studies is the ability to understand the behavior of statistical methods because some "truth" (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analyzed, and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting, and presentation. In particular, this tutorial provides a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods, and performance measures ("ADEMP"); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine, which included at least one simulation study and identify areas for improvement.
Citations
Cited by
- Blinded‐Into‐Unblinded Interim Analyses for Clinical Trials With Time‐to‐Event Endpoints
- Simulation-Free Bayesian Power and Sample Size Calculations for Bayes Factors in Single-Arm Phase II Trials with Binary Endpoints
- Quantifying potential selection bias in observational research: simulations and Analyses exploring religion and depression using a prospective UK cohort study (ALSPAC)
- Longitudinal Outcomes Truncated by Death: Causal Estimands and Bayesian Estimators
- Hidden multistate models to study multimorbidity trajectories
- Data-driven controlled subgroup selection in clinical trials
- Exploratory Mean-Variance with Jumps: An Equilibrium Approach
- Network Meta Analysis of Mean Survival
- Novel g-computation algorithms for time-varying actions with recurrent and semi-competing events
- Getting it right: Methods for risk ratios and risk differences cluster randomized trials with a small number of clusters
- The Bayesian optimal two-stage design for clinical phase II trials based on Bayes factors
- Can discrete-time analyses be trusted for stepped wedge trials with continuous recruitment?
- Balancing Evidentiary Value and Sample Size of Adaptive Designs with Application to Animal Experiments
- Simulating Data From Marginal Structural Models for a Survival Time Outcome
- An Agent-Based Simulation of Regularity-Driven Student Attrition: How Institutional Time-to-Live Constraints Create a Dropout Trap in Higher Education
- Is inverse probability of censoring weighting a safer choice than per-protocol analysis in clinical trials?
- Comparative effectiveness research considered methodological insights from simulation studies in physician's prescribing preference
- Inverse Probability Weighting of Count Exposures in the Presence of Missing Data: A Simulation Study
- Mean-Tilted Relaxed Quantile Regression: Fixed-Content Interval Functionals and Generalized-Bayes Computation
- Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method
- Toward a standardized evaluation of imputation methodology
- Recommendations for temporal aggregation of water quality data from multi-platform satellite constellations
- Pharmacoepidemiology simulation study practices: A methodological review
- Power calculation for cross-sectional stepped wedge cluster randomized trials with a time-to-event endpoint
- Causal Effect Estimation with TMLE: Handling Missing Data and Near-Violations of Positivity
- Optimal weighted tests for replication studies and the two-trials rule
- Inverse-intensity weighted generalized estimating equations with irregularly measured longitudinal data and informative dropout
- Mendelian randomization
- Detecting departures from the conditional independence assumption in diagnostic latent class models: a simulation study
- Ten simple rules for reporting machine learning methods implementation and evaluation on biomedical data
- Edgington's Method for Random-Effects Meta-Analysis Part I: Estimation
- A comparison of approaches to incorporate patient-selected and patient-ranked outcomes in clinical trials
- Replicability of simulation studies for the investigation of statistical methods: the RepliSims project
- Stabilizing Thompson Sampling with Point Null Bayesian Response-Adaptive Randomization
- Sample-Size Planning in Item-Response Theory: A Tutorial
- Unbiased Estimates Using Temporally Aggregated Outcome Data in Time Series Analysis: Generalization to Different Outcomes, Exposures, and Types of Aggregation
- Bridging the Gap Between Methodological Research and Statistical Practice: Toward "Translational Simulation Research
- Selection Bias in Hybrid Randomized Controlled Trials using External Controls: A Simulation Study
- Using directed acyclic graphs to determine whether multiple imputation or subsample-multiple imputation estimates of an exposure-outcome association are unbiased
- The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
- Value-adaptive clinical trial designs for efficient delivery of publicly funded trials - a discussion of methods, case studies, opportunities and challenges
- Towards more appropriate modelling of linguistic complexity measures: Beyond traditional regression models
- Declaring and Diagnosing Research Designs
- Many nonnormalities, one simulation: Do different data generation algorithms affect study results?
- Toward a rigorous assessment of the statistical performances of methods to estimate the Minimal Important Difference of Patient-Reported Outcomes: A protocol for a large-scale simulation study
- Addressing Missing Data in Accelerometer Studies: Evaluating the Performance of Imputation Methods for Longitudinal Data
- Evaluating the sample size requirements of tree-based ensemble machine learning techniques for clinical risk prediction
- Composition of core modules and item allocation in split questionnaire designs: impact on estimates from imputed data
- On variance estimation of the inverse probability‐of‐treatment weighting estimator: A tutorial for different types of propensity score weights
- Categorisation of continuous covariates for stratified randomisation: How should we adjust?
- Adapting Nearest Neighbor for Multiple Imputation: Advantages, Challenges, and Drawbacks
- Machine learning-guided computational discovery of anti-ovarian cancer leads from fungal secondary metabolites
- Efficient interaction analysis in randomized controlled trials
- Optimizing sPESI with heart rate threshold adjustments for risk stratification in acute pulmonary embolism: A retrospective cohort study
- Simulating Complex Cross-Sectional and Longitudinal Data Using the simDAG R Package
- Enhanced Lepage-type test statistics for location-scale shifts with right-skewed data
- Estimation and inference in generalised linear models with constrained iteratively-reweighted least squares
- Minimum Sample Size Calculation for Multivariable Regression of Continuous Outcomes in Chemometrics for Astrobiology and Planetary Science
- Tidy simulation: Designing robust, reproducible, and scalable Monte Carlo simulations
- A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint
- Adaptive clinical trial design with delayed treatment effects using elicited prior distributions
- A regularized multi-state model for covariate selection with interval-censored survival data
- Examining the Association between Estimated Prevalence and Diagnostic Test Accuracy using Directed Acyclic Graphs
- Design and analysis of individually randomized multiple baseline factorial trials
- Covariate adjustment for linear models in estimating treatment effects in randomised clinical trials. Some useful theory to guide simulation
- The Win Ratio at the Design Stage of Clinical Trials
- Risk-inclusive Contextual Bandits for Early Phase Clinical Trials
- Evaluating the effect of different non-informative prior specifications on the Bayesian proportional odds model in randomised controlled trials: a simulation study
- Permutation Tests Based on the Copula-Graphic Estimator and Their Use for Survival Tree Construction
- Is 1:1 Always Most Powerful? Why Careful Determination of Allocation Ratios Matters in Trial Design
- Network meta-analysis of rare events using penalized likelihood regression
- Inference procedures in sequential trial emulation with survival outcomes: comparing confidence intervals based on the sandwich variance estimator, bootstrap and jackknife
Discussions
Related