Robust benchmarking in noisy environments

2016/08/15 by Jiahao Chen, Chen, Jiahao, Jarrett Revels +1 · 1 voice · 17 citations
Computer Science · #68N30 #Advanced Data Storage Technologies #B.8.1 #D.2.5 #Embedded Systems Design Techniques #FOS: Computer and information sciences #Parallel Computing and Optimization Techniques #Performance (cs.PF) #acm:68N30 #cs.PF #msc:68N30

paper · pdf · doi:10.48550/arxiv.1608.04295

7 pages, 5 figures, Proceedings of the 20th Annual IEEE High Performance Extreme Computing Conference, 2016

arxiv created 2016/08/15 · openalex publication_date 2016/08/15 · arxiv published 2016/08/15 · arxiv updated 2016/08/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.

Citations

• Virtual Machine Warmup Blows Hot and Cold

Cited by

• A fast, large-scale optimal transport algorithm for holographic beam shaping
• Flux-Preserving Adaptive Finite State Projection for Multiscale Stochastic Reaction Networks
• Optimizations on Graph-Level for Domain Specific Computations in Julia and Application to QED
• SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis
• Rethinking PCA Through Duality
• Discriminating Equivalent Algorithms via Relative Performance
• Reciprocal Multi-Robot Collision Avoidance with Asymmetric State Uncertainty
• Correlations between Dust Extinction Features across All Wavelength Scales: From Diffuse Interstellar Bands to R(V)
• On circumcentered direct methods for monotone variational inequality problems
• Explicit Density Approximation for Neural Implicit Samplers Using a Bernstein-Based Convex Divergence
• Symbolic Model Checking in External Memory
• Efficient Calculation of Modified Bessel Functions of the First Kind, I_ν (z), for Real Orders and Complex Arguments: Fortran Implementation with Double and Quadruple Precision
• Scalable Exact Inference in Multi-Output Gaussian Processes
• The Right Call for Software Benchmarking: Consistent Decisions in Stateful Environments
• Constructing Magic Squares: an integer constraint satisfaction problem and a fast approach
• FormulaCompiler.jl and Margins.jl: Efficient Marginal Effects in Julia
• ChromOps.jl: High-order simulation and discrete forward sensitivity analysis for chromatography models

Discussions

• Robust benchmarking in noisy environments [hn, 2 points, 0 comments]

Related

• The Right Call for Software Benchmarking: Consistent Decisions in Stateful Environments
• µOpTime: Statically reducing the execution time of microbenchmark suites using stability metrics
• μOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics
• Benchmarking Blunders and Things That Go Bump in the Night
• SILENT: A New Lens on Statistics in Software Timing Side Channels