nbodykit: an open-source, massively parallel toolkit for large-scale structure

2017/12/15 by Nick Hand, Yu Feng, Florian Beutler +5 · 255 citations
Biochemistry, Genetics and Molecular Biology · Materials Science · Mathematics · Physics and Astronomy · #Data structure #Implementation #Machine Learning in Materials Science #Massively parallel #Protein Structure and Dynamics #Python (programming language) #Scalability #Supercomputer #Tensor decomposition and applications #astro-ph.CO #astro-ph.IM

paper · pdf · open access · doi:10.3847/1538-3881/aadae0

published in The Astronomical Journal 156(4), 160 (Institute of Physics) · 18 pages, 7 figures. Feedback very welcome. Code available at https://github.com/bccp/nbodykit and for documentation, see http://nbodykit.readthedocs.io

arxiv created 2017/12/15 · openalex created_date 2018/01/05 · openalex publication_date 2018/09/18 · arxiv updated 2018/10/31 · openalex updated_date 2026/08/05

Abstract

We present nbodykit, an open-source, massively parallel Python toolkit for analyzing large-scale structure (LSS) data. Using Python bindings of the Message Passing Interface (MPI), we provide parallel implementations of many commonly used algorithms in LSS. nbodykit is both an interactive and scalable piece of scientific software, performing well in a supercomputing environment while still taking advantage of the interactive tools provided by the Python ecosystem. Existing functionality includes estimators of the power spectrum, 2 and 3-point correlation functions, a Friends-of-Friends grouping algorithm, mock catalog creation via the halo occupation distribution technique, and approximate N-body simulations via the FastPM scheme. The package also provides a set of distributed data containers, insulated from the algorithms themselves, that enable nbodykit to provide a unified treatment of both simulation and observational data sets. nbodykit can be easily deployed in a high performance computing environment, overcoming some of the traditional difficulties of using Python on supercomputers. We provide performance benchmarks illustrating the scalability of the software. The modular, component-based approach of nbodykit allows researchers to easily build complex applications using its tools. The package is extensively documented at http://nbodykit.readthedocs.io, which also includes an interactive set of example recipes for new users to explore. As open-source software, we hope nbodykit provides a common framework for the community to use and develop in confronting the analysis challenges of future LSS surveys.

Citations

• Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant

Cited by

• How Do Uncertainties in Galaxy Formation Physics Impact Field-level Galaxy Bias?
• Density reconstruction from biased tracers: Testing the equivalence principle through consistency relations
• Reviving sub-keV warm dark matter: a UVLF-based analysis
• Investigating the Reionization Epoch through 21 cm and Line Intensity Mapping Experiments
• Clues from Q--A null test designed for line intensity mapping cross-correlation studies
• Primordial non-Gaussianity -- Fast simulations and persistent summary statistics
• Probing Cosmic Magnetism with Rotation Measure-Squared-Galaxy Cross-Correlations
• Baryonic Feedback across Halo Mass: Impact on the Matter Power Spectrum
• First Cosmological Constraints from the Joint Analysis of Galaxy Clustering and the Kinetic Sunyaev-Zel'dovich Effect
• Bias from small-scale leakage in Pulsar Timing Array maps
• Measurement of Parity-Violating Modes of the Dark Energy Spectroscopic Instrument (DESI) Year 1 Luminous Red Galaxies' 4-Point Correlation Function
• Power Spectrum, Bispectrum, 2- and 3-Point Correlation Function, and Beyond
• Warm Dark Matter meets Cold Dark Matter Isocurvature
• Effective Theories of Redshift-Space Galaxy Peculiar Velocities
• Cosmic curl -- Features and convergence of the vorticity power spectrum in N-body simulations
• Cosmological Remapping for Efficient Generation of 21 cm Intensity Mapping Mocks
• Modeling Gravitational Wave Bias from 3D Power Spectra of Spectroscopic Surveys
• Probing cosmic isotropy in the Local Universe
• Local primordial non-Gaussianity from 'zero-bias' 21cm radiation during reionization
• Testing the general theory of relativity using gravitational wave propagation from dark standard sirens
• Accurate initial conditions for cosmological N-body simulations: minimizing truncation and discreteness errors
• A Hybrid Deep Learning Approach to Cosmological Constraints from Galaxy Redshift Surveys
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: 1000 multi-tracer mock catalogues with redshift evolution and systematics for galaxies and quasars of the final data release
• Impact of cosmological signatures in two-point statistics beyond the linear regime
• Modeling galaxies in redshift space at the field level
• Building a digital twin of a luminous red galaxy spectroscopic survey: galaxy properties and clustering covariance
• The global environment of small galaxy systems
• Clustering in the simulated H α galaxy redshift survey from Nancy Grace Roman Space Telescope
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: N-body mock challenge for the quasar sample
• Reducing the variance of redshift space distortion measurements from mock galaxy catalogues with different lines of sight
• Baryon acoustic oscillations reconstruction using convolutional neural networks
• Enhancing DESI DR1 Full-Shape analyses using HOD-informed priors
• Likelihood non-Gaussianity in large-scale structure analyses

Related

• Basic Data Analysis and More - A Guided Tour Using Python
• Hermes - Towards an Optimal High-Performance Algorithm for Cosmic Statistics of Large Data Sets
• PolyBin3D: A Suite of Optimal and Efficient Power Spectrum and Bispectrum Estimators for Large-Scale Structure
• Building a scalable python distribution for HEP data analysis
• Analyzing billion-objects catalog interactively: Apache Spark for physicists