2018/07/04 by Umberto Ferraro Petrillo, Petrillo, Umberto Ferraro, Mara Sorella +7 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Advanced Data Storage Technologies #Algorithms and Data Compression #Distributed #FOS: Computer and information sciences #Gene expression and cancer classification #Genomics and Phylogenetic Studies #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1807.01566
openalex publication_date 2018/07/04 · openalex created_date 2022/08/04 · openalex updated_date 2026/07/28
Distributed approaches based on the map-reduce programming paradigm have\nstarted to be proposed in the bioinformatics domain, due to the large amount of\ndata produced by the next-generation sequencing techniques. However, the use of\nmap-reduce and related Big Data technologies and frameworks (e.g., Apache\nHadoop and Spark) does not necessarily produce satisfactory results, in terms\nof both efficiency and effectiveness. We discuss how the development of\ndistributed and Big Data management technologies has affected the analysis of\nlarge datasets of biological sequences. Moreover, we show how the choice of\ndifferent parameter configurations and the careful engineering of the software\nwith respect to the specific framework under consideration may be crucial in\norder to achieve good performance, especially on very large amounts of data. We\nchoose k-mers counting as a case study for our analysis, and Spark as the\nframework to implement FastKmer, a novel approach for the extraction of k-mer\nstatistics from large collection of biological sequences, with arbitrary values\nof k. One of the most relevant contributions of FastKmer is the introduction of\na module for balancing the statistics aggregation workload over the nodes of a\ncomputing cluster, in order to overcome data skew while allowing for a fully\nexploitation of the underly- ing distributed architecture. We also present the\nresults of a comparative experimental analysis showing that our approach is\ncurrently the fastest among the ones based on Big Data technologies, while\nexhibiting a very good scalability. We provide evidence that the usage of\ntechnologies such as Hadoop or Spark for the analysis of big datasets of\nbiological sequences is productive only if the architectural details and the\npeculiar aspects of the considered framework are carefully taken into account\nfor the algorithm design and implementation.\n