arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.00942cs.DC
收藏

通过大数据Spark平台进行免比对基因组分析

Alignment-free Genomic Analysis via a Big Data Spark Platform

Umberto Ferraro Petrillo, Francesco Palini, Giuseppe Cattaneo, Raffaele Giancarlo

摘要:

动机:无比对距离和相似性函数(简称AF函数)是许多基因组、宏基因组和表观基因组任务中两个和多个序列比对的公认替代方案。由于数据密集型应用,AF函数的计算是一个大数据问题,最近的文献表明,快速和可扩展的算法计算AF函数的发展是一个高优先级的任务。有些令人惊讶的是,尽管大数据技术在计算生物学中越来越受欢迎,但针对这些任务的大数据平台的开发尚未得到追求,可能是由于其复杂性。结果如下:我们通过引入FADE填补了这一重要空白,FADE是第一个可扩展,高效和可扩展的Spark平台,用于无比对基因组分析。它支持最近一项标志性基准研究中的18项最佳AF功能。FADE的发展和潜在影响包括感兴趣的新方面。也就是说,(a)分布式算法的相当大的努力,最切实的结果是MASH和FSWM等参考方法的执行时间更快;(b)软件设计,使FADE用户友好,易于Spark非专家扩展;(c)支持数据和计算密集型任务的能力。关于这一点,我们提供了一个新的和急需的分析如何信息和强大的AF功能,在其输出的统计意义。我们的发现自然扩展了备受推崇的基准研究,因为真正可以使用的功能减少到FADE中包含的18个功能中的少数几个。

英文摘要:

Motivation: Alignment-free distance and similarity functions (AF functions, for short) are a well established alternative to two and multiple sequence alignments for many genomic, metagenomic and epigenomic tasks. Due to data-intensive applications, the computation of AF functions is a Big Data problem, with the recent Literature indicating that the development of fast and scalable algorithms computing AF functions is a high-priority task. Somewhat surprisingly, despite the increasing popularity of Big Data technologies in Computational Biology, the development of a Big Data platform for those tasks has not been pursued, possibly due to its complexity. Results: We fill this important gap by introducing FADE, the first extensible, efficient and scalable Spark platform for Alignment-free genomic analysis. It supports natively eighteen of the best performing AF functions coming out of a recent hallmark benchmarking study. FADE development and potential impact comprises novel aspects of interest. Namely, (a) a considerable effort of distributed algorithms, the most tangible result being a much faster execution time of reference methods like MASH and FSWM; (b) a software design that makes FADE user-friendly and easily extendable by Spark non-specialists; (c) its ability to support data- and compute-intensive tasks. About this, we provide a novel and much needed analysis of how informative and robust AF functions are, in terms of the statistical significance of their output. Our findings naturally extend the ones of the highly regarded benchmarking study, since the functions that can really be used are reduced to a handful of the eighteen included in FADE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑