2019/12/02 by Jianfeng Zhan, Zhan, Jianfeng, Lei Wang +5 · 1 citation
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Cloud Computing and Resource Management #FOS: Computer and information sciences #Performance (cs.PF) #Scientific Computing and Data Management #Software System Performance and Reliability #cs.AI #cs.PF
paper · pdf · doi:10.48550/arxiv.1912.00572
7 pages
openalex publication_date 2019/12/02 · arxiv created 2019/12/03 · arxiv updated 2019/12/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This paper outlines BenchCouncil's view on the challenges, rules, and vision of benchmarking modern workloads like Big Data, AI or machine learning, and Internet Services. We conclude the challenges of benchmarking modern workloads as FIDSS (Fragmented, Isolated, Dynamic, Service-based, and Stochastic), and propose the PRDAERS benchmarking rules that the benchmarks should be specified in a paper-and-pencil manner, relevant, diverse, containing different levels of abstractions, specifying the evaluation metrics and methodology, repeatable, and scaleable. We believe proposing simple but elegant abstractions that help achieve both efficiency and general-purpose is the final target of benchmarking in future, which may be not pressing. In the light of this vision, we shortly discuss BenchCouncil's related projects.