Sustainable data analysis with Snakemake
2021/04/19 by Felix Mölder, Kim Philipp Jablonski, Brice Letcher +17 · 79 citations
Computer Science · Decision Sciences · #Data Quality and Management #Research Data Management Practices #Scientific Computing and Data Management
paper · pdf · doi:10.12688/f1000research.29032.2
openalex created_date 2021/02/01 · openalex publication_date 2021/04/19 · openalex updated_date 2026/07/30
Abstract
Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.
Citations
Cited by
- Microbiology Galaxy Lab: The first community-driven gateway for reproducible and FAIR analysis of microbial data
- A Pedigree-Based Map of Crossovers and Noncrossovers in Aye-Ayes ( <i>Daubentonia madagascariensis</i> )
- Cohesin guides homology search during DNA repair via loops and sister chromatid linkages
- Metagenomic time series reveals a Western English Channel viral community dominated by members with strong seasonal signals
- Enhancing metabarcoding efficiency and ecological insights through integrated taxonomy and DNA reference barcoding: A case study on beach meiofauna
- Revisiting adaptive introgression at the HLA genes in Lithuanian genomes with machine learning
- cfDNA UniFlow: a unified preprocessing pipeline for cell-free DNA data from liquid biopsies
- The Peptonizer2000: Bringing Confidence to Metaproteomics
- Building a FAIR image data ecosystem for microscopy communities
- The Genomics of Convergent Adaptation to Intertidal Gravel Beaches in Mediterranean Clingfishes
- Advanced analysis of retrotransposon variation in the human genome with nanopore sequencing using RetroInspector
- BUSCO: Assessing Genomic Data Quality and Beyond
- AOC: Analysis of Orthologous Collections -- an application for the characterization of natural selection in protein-coding sequences
- Long-read metagenomics and methylation-based binning allow the description of the emerging high-risk antibiotic resistance genes and their hidden hosts in complex communities
- Early-life viability selection targets deleterious mutations in exons
- SmallBARNA 2026: a kingdom-wide bacterial sRNA resource
- Association between SARS-CoV-2 and metagenomic content of samples from the Huanan Seafood Market
- Dairy cows inoculated with highly pathogenic avian influenza virus H5N1
- Pooled, Long-read Sequencing for Structural Variant Characterization in Schistosome Populations
- Characterization of the transposable element landscape shaping the Ectocarpus genome
- Tracing the evolutionary history of the CCR5delta32 deletion via ancient and modern genomes
- Theory of Field-Dependent NMR Shifts in Paramagnetic Molecules
- SAI: A Python Package for Statistics for Adaptive Introgression
- High intra-laboratory reproducibility of nanopore sequencing in bacterial species underscores advances in its accuracy
- Guidance framework to apply best practices in ecological data analysis: lessons learned from building Galaxy-Ecology
- Fluke-borne viruses are a risk factor for diseases caused by carcinogenic trematodes
- Near real-time monitoring of wading birds using uncrewed aircraft systems and computer vision
- EESSI: A cross‐platform ready‐to‐use optimised scientific software stack
- RiboSnake – a user-friendly, robust, reproducible, multipurpose and documentation-extensive pipeline for 16S rRNA gene microbiome analysis
- A call for clean code to effectively communicate science
- Long-Read–Based Genome Assembly Reveals Numerous Endogenous Viral Elements in the Green Algal Bacterivore <i>Cymbomonas tetramitiformis</i>
- Iron limitation promotes metabolic cross-feeding between cheese ripening bacteria
- Gene Expression Patterns Regulating Peanut Reproductive Phenology
- Stress-induced ribosome degradation in <i>Bacillus subtilis</i> is mediated by the RNase Y-specificity complex
- The Molecular and Functional Landscape of Resistance to FOLFIRI Chemotherapy in Metastatic Colorectal Cancer
- BOTANIC-0: a series of foundation models for plant genomic data
- Strand-resolved mutagenicity of DNA damage and repair
- Inflammatory stromal and T cells mediate human bone marrow niche remodeling in clonal hematopoiesis and myelodysplasia
- <scp> <i>detectEVE</i> </scp> : Fast, Sensitive and Precise Detection of Endogenous Viral Elements in Genomic Data
- The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update
- Coordinated repression of totipotency-associated gene loci by histone methyltransferase EHMT2 via LINE1 regulatory elements
- HaploCharmer: a Snakemake workflow for read-scale haplotype calling adapted to polyploids
- The Data are Insufficient to Confidently Root the SARS-CoV-2 Phylogenetic Tree
- scTAM-seq enables targeted high-confidence analysis of DNA methylation in single cells
- Genomic basis of adaptation to serpentine soil in two <i>Alyssum</i> species shows convergence with <i>Arabidopsis</i> across 20 million years of divergence
- Disruption of the mRNA m <sup>6</sup> A writer complex triggers autoimmunity in Arabidopsis
- A single gene orchestrates androgen variation underlying male mating morphs in ruffs
- Functional gut microbiota dynamics of generalist and specialist bacteria in association with chicken growth
- HAPP: High-accuracy pipeline for processing deep metabarcoding data
- Focal white matter lesions drive grey matter inflammation and synapse loss
- A seven‐step guide to spatial, agent‐based modelling of tumour evolution
- Adaptive adjustment of profile HMM significance thresholds improves functional and metabolic insights into microbial genomes
- Quantitative ethology of schistosome miracidia characterizes a conserved snail peptide that inhibits host recognition
- Dupilumab Treatment Is Associated With Clinical Improvement and a Shift Toward a Health‐Associated Nasal Passage Microbiota in Diffuse Type 2 Chronic Rhinosinusitis
- Proteomic analysis of dental enamel from 20 Homo naledi individuals shows no male markers
- Generalization of Bleaney’s Theory
- Komplexe Tabellen verständlich machen
- Assessing species coverage and assembly quality of rapidly accumulating sequenced genomes
- HiFIseek: gene-specific enrichment of high-impact mutations in associated genomic regions
- Nanorate sequencing reveals the <i>Arabidopsis</i> somatic mutation landscape
- Evaluation of accessibility disparities in urban areas during disruptive events based on transit real data
- Assessing genomic reproducibility of read alignment tools
- Peritoneal resident macrophages constitute an immunosuppressive environment in peritoneal metastasized colorectal cancer
- Generalization of the sci-L3 method to achieve high-throughput linear amplification for replication template strand sequencing, genome conformation capture, and the joint profiling of RNA and chromatin accessibility
- Induced selection of tebuconazole-resistant Aspergillus flavus isolates during germination of treated corn seeds
- A constitutive interferon-high immunophenotype defines response to immunotherapy in colorectal cancer
- Exploring DNA degradation in situ and in museum storage through genomics and metagenomics
- Pangenome databases improve host removal and mycobacteria classification from clinical metagenomic data
- Spyglass: a framework for reproducible and shareable neuroscience research
- BioImageIT: A novel python‐based architecture for reproducible bio‐image workflows
- From constraint to opportunity: Relaxing sexual antagonism reveals adaptive potential maintained by balancing selection
- Do we still need Illumina sequencing data? Evaluating Oxford Nanopore Technologies R10.4.1 flow cells and the Rapid v14 library prep kit for Gram negative bacteria whole genome assemblies
- Ten simple rules to follow when cleaning occurrence data in palaeobiology
- Genome-resolved long-read sequencing expands known microbial diversity across terrestrial habitats
- Over 1000 tools reveal trends in the single-cell RNA-seq analysis landscape
- LabChain: Enabling reproducible and modular scientific experiments in Python
- Recovering signatures of archaic hominin introgression using ancestral recombination graphs
- A functional interaction between TDP-43 and USP10 reveals USP10 dysfunction in TDP-43 proteinopathies
- Polygenic and monogenic adaptation drive evolutionary rescue at different magnitudes of environmental change
Related