A Ceph S3 Object Data Store for HEP

2023/11/27 by Nick Smith, Smith, Nick, B. Jayatilaka +11 · 1 citation
Computer Science · Decision Sciences · #Advanced Data Storage Technologies #Data Analysis #Distributed and Parallel Computing Systems #FOS: Physical sciences #High Energy Physics - Experiment (hep-ex) #Scientific Computing and Data Management #Statistics and Probability (physics.data-an)

paper · pdf · doi:10.48550/arxiv.2311.16321

openalex publication_date 2023/11/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.

Cited by

• OASIS: Object-based Analytics Storage for Intelligent SQL Query Offloading in Scientific Tabular Workloads

Related

• The Event as an Object-Relational Database: Avoiding the Dependency Nightmare
• Persistent Data Layout and Infrastructure for Efficient Selective Retrieval of Event Data in ATLAS
• Toward real-time data query systems in HEP
• RNTuple: Towards First-Class Support for HPC data centers
• Next generation input-output data format for HEP using Google's protocol buffers