arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1606.06204cs.DCcs.DS
收藏

分布式集群上三角网格数字高程模型的并行优先级-洪降填充

Parallel Priority-Flood Depression Filling For Trillion Cell Digital Elevation Models On Desktops Or Clusters

Richard Barnes

摘要:

从数字高程模型(DEM)中提取水文特征和属性的算法受到大型数据集的挑战,这些数据集通常无法容纳在计算机的RAM中。凹陷填充是许多这些算法的重要预处理步骤。在这里,我提出了一个新的,线性缩放算法,并行的优先级洪水凹陷填充算法细分成瓷砖的DEM。使用单生产者,多消费者的设计,新算法在一个核心,多个核心或多台机器上同样有效,可以利用大内存或处理小内存。与以前的算法不同,新算法保证了一个固定数量的内存访问和通信事件的每个细分的DEM。在比较测试中,这导致新算法运行速度一般更快,同时使用更少的资源比以前的算法。对于中等大小的瓦片,该算法表现出~60%的强和弱缩放效率高达48个核心,以及跨越三个数量级的数据集的线性时间缩放。我运行该算法的最大数据集有2万亿(2*10^12)个细胞。对于48个核心,处理需要4.8小时的壁时间(9.3计算日)。该测试比文献中先前进行的任何测试大三个数量级。完整的、注释良好的源代码和正确性测试可以从存储库下载。

英文摘要:

Algorithms for extracting hydrologic features and properties from digital elevation models (DEMs) are challenged by large datasets, which often cannot fit within a computer's RAM. Depression filling is an important preconditioning step to many of these algorithms. Here, I present a new, linearly-scaling algorithm which parallelizes the Priority-Flood depression-filling algorithm by subdividing a DEM into tiles. Using a single-producer, multi-consumer design, the new algorithm works equally well on one core, multiple cores, or multiple machines and can take advantage of large memories or cope with small ones. Unlike previous algorithms, the new algorithm guarantees a fixed number of memory access and communication events per subdivision of the DEM. In comparison testing, this results in the new algorithm running generally faster while using fewer resources than previous algorithms. For moderately sized tiles, the algorithm exhibits ~60% strong and weak scaling efficiencies up to 48 cores, and linear time scaling across datasets ranging over three orders of magnitude. The largest dataset on which I run the algorithm has 2 trillion (2*10^12) cells. With 48 cores, processing required 4.8 hours wall-time (9.3 compute-days). This test is three orders of magnitude larger than any previously performed in the literature. Complete, well-commented source code and correctness tests are available for download from a repository.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑