Extreme Scale De Novo Metagenome Assembly

Checkpointing SPAdes for Metagenome Assembly: Transparency versus Performance in Production

The SPAdes assembler for metagenome assembly is a long-running application commonly used at the NERSC supercomputing site. However, NERSC, like many other sites, has a 48-hour limit on resource allocations. The solution is to chain together multiple resource allocations in a single run, using checkpoint-restart. This case study provides insights into the "pain points" in applying a well-known checkpointing package (DMTCP: Distributed MultiThreaded CheckPointing) to long-running production workloads of SPAdes. This work has exposed several bugs and limitations of DMTCP, which were fixed to supp…

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

One of the most computationally intensive tasks in computational biology is de novo genome assembly, the decoding of the sequence of an unknown genome from redundant and erroneous short sequences. A common assembly paradigm identifies overlapping sequences, simplifies their layout, and creates consensus. Despite many algorithms developed in the literature, the efficient assembly of large genomes is still an open problem. In this work, we introduce new distributed-memory parallel algorithms for overlap detection and layout simplification steps of de novo genome assembly, and implement them in t…

Distributed-Memory Parallel Contig Generation for De Novo Long-Read Genome Assembly

De novo genome assembly, i.e., rebuilding the sequence of an unknown genome from redundant and erroneous short sequences, is a key but computationally intensive step in many genomics pipelines. The exponential growth of genomic data is increasing the computational demand and requires scalable, high-performance approaches. In this work, we present a novel distributed-memory algorithm that, from a string graph representation of the genome and using sparse matrices, generates the contig set, i.e., overlapping sequences that form a map representing a region of a chromosome. Using matrix abstractio…

NMP-PaK：可扩展从头基因组组装的近内存处理加速

NMP-PaK: Near-Memory Processing Acceleration of Scalable De Novo Genome Assembly

本文提出软硬件协同设计 NMP-PaK，利用通道级近内存处理、定制处理单元、批处理和 CPU-NMP 混合执行加速可扩展从头基因组组装，显著降低内存占用并提升性能与吞吐量。

Scalable De Novo Genome Assembly Using Pregel

De novo genome assembly is the process of stitching short DNA sequences to generate longer DNA sequences, without using any reference sequence for alignment. It enables high-throughput genome sequencing and thus accelerates the discovery of new genomes. In this paper, we present a toolkit, called PPA-assembler, for de novo genome assembly in a distributed setting. The operations in our toolkit provide strong performance guarantees, and can be assembled to implement various sequencing strategies. PPA-assembler adopts the popular {\em de Bruijn graph} based approach for sequencing, and each oper…

极端规模的从头基因组组装

Extreme-Scale De Novo Genome Assembly

从头全基因组组装是从短的、重叠的和可能错误的DNA片段重建基因组序列，是现代基因组学中最重要的计算之一。这项工作提出了HipMER，一个高品质的端到端的从头汇编设计的极端规模分析，通过有效的并行化的Mercury代码。基因组组装软件有许多组件，每个组件强调计算机系统的不同组件。本章详细解释了HipMer流水线的每一步所涉及的计算挑战、关键的分布式数据结构和通信成本。我们目前的性能结果组装的人类基因组和大型六倍体小麦基因组的大型超级计算机高达数万个核心。