基于MapReduce的后缀数组可扩展高效构造及内存数据存储系统
Scalable and Efficient Construction of Suffix Array with MapReduce and In-Memory Data Store System
摘要：
后缀数组（SA）是模式匹配的基本数据结构，在数据压缩、剽窃检测和序列比对等领域有着广泛的应用。然而，随着数据量的急剧增加，SA的构建不再适合当前的大规模数据处理框架，这是由于其在构建过程中固有的后缀扩散。也就是说，仅仅通过向框架添加资源来改善性能变得成本效益较低，甚至具有严重的收益递减。现在的问题是，我们是否可以通过创造一个彻底的视角转变来允许SA构建更具可扩展性和效率，以实现数据的永久增长。关于TeraSort [1]作为我们的基线，我们首先展示了TeraSort脆弱的可扩展性，并通过石斑鱼序列比对实验（即，生物信息学中使用的SA识别）。因此，我们提出了一个方案，合并到MapReduce的分布式键值存储系统，利用内存中的查询后缀。MapReduce不是处理后缀的通信，而是负责它们的索引的通信，这意味着更好的数据容量。它大大减少了构建SA所需的磁盘空间，并更好地利用内存，从而从根本上提高了可伸缩性。我们还研究了我们的计划在内存方面的效率，并显示它优于TeraSort。最后，我们的方案可以在不降低可扩展性的情况下，用两个输入文件完成对端排序和对齐，并且可以在16个节点和千兆以太网组成的小型集群中容纳近6.7 TB的后缀，而不需要任何压缩。
英文摘要：
Suffix Array (SA) is a cardinal data structure in many pattern matching applications, including data compression, plagiarism detection and sequence alignment. However, as the volumes of data increase abruptly, the construction of SA is not amenable to the current large-scale data processing frameworks anymore due to its intrinsic proliferation of suffixes during the construction. That is, ameliorating the performance by just adding the resources to the frameworks becomes less cost- effective, even having the severe diminishing returns. At issue now is whether we can permit SA construction to be more scalable and efficient for the everlasting accretion of data by creating a radical shift in perspective. Regarding TeraSort [1] as our baseline, we first demonstrate the fragile scalability of TeraSort and investigate what causes it through the experiments on the sequence alignment of a grouper (i.e., the SA construc- tion used in bioinformatics). As such, we propose a scheme that amalgamates the distributed key-value store system into MapReduce to leverage the in-memory queries about suffixes. Rather than handling the communication of suffixes, MapReduce is in charge of the communication of their indexes, which means better capacity for more data. It significantly abates the required disk space for constructing SA and better utilizes the memory, which in turn improves the scalability radically. We also examine the efficiency of our scheme in terms of memory and show it outperforms TeraSort. At last, our scheme can complete the pair- end sequencing and alignment with two input files without any degradation on scalability, and can accommodate the suffixes of nearly 6.7 TB in a small cluster composed of 16 nodes and Gigabit Ethernet without any compression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
