arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.01566cs.DC
收藏

基因组序列大数据集分析:快速可扩展的k-mer统计量收集

Analyzing Big Datasets of Genomic Sequences: Fast and Scalable Collection of k-mer Statistics

Umberto Ferraro Petrillo, Mara Sorella, Giuseppe Cattaneo, Raffaele Giancarlo, Simona Rombo

摘要:

由于下一代测序技术产生的大量数据,基于映射-缩减编程范式的分布式方法已经开始在生物信息学领域被提出。然而,使用map-reduce和相关的大数据技术和框架(例如,Apache Hadoop和Spark)在效率和有效性方面不一定会产生令人满意的结果。我们讨论了分布式和大数据管理技术的发展如何影响了生物序列的大数据集的分析。此外,我们展示了如何选择不同的参数配置和仔细工程的软件相对于所考虑的特定框架可能是至关重要的,以实现良好的性能,特别是在非常大量的数据。我们选择k-mer计数作为我们分析的案例研究,并Spark作为实现FastKmer的框架,FastKmer是一种从大量生物序列中提取k-mer统计的新方法,具有任意k值。FastKmer最相关的贡献之一是引入了一个模块,用于在计算集群的节点上平衡统计数据聚合工作负载,以克服数据偏斜,同时允许充分利用底层分布式架构。我们还提出了一个比较实验分析的结果,表明我们的方法是目前最快的基于大数据技术,同时表现出非常好的可扩展性。我们提供的证据表明,使用技术,如Hadoop或Spark的生物序列的大数据集的分析是富有成效的,只有当架构的细节和所考虑的框架的特殊方面仔细考虑到的算法设计和实施。

英文摘要:

Distributed approaches based on the map-reduce programming paradigm have started to be proposed in the bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of map-reduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose k-mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of k-mer statistics from large collection of biological sequences, with arbitrary values of k. One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a fully exploitation of the underly- ing distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability. We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑