科学计算与大数据技术的结合：天文学应用案例
Scientific Computing Meets Big Data Technology: An Astronomy Use Case
摘要：
科学分析通常将多个单进程程序组合成一个子程序。单进程程序的端到端并行流被称为多任务应用程序。通常，来自HPC软件栈的工具用于并行化这些分析。在这项工作中，我们研究了一种替代方法，该方法使用Apache Spark（一种现代大数据平台）来并行化多任务应用程序。我们提出Kira，一个灵活的分布式天文学图像处理工具包，使用Apache Spark。然后，我们使用Kira工具包来实现天文图像的源提取器应用程序，称为Kira SE。以Kira SE为用例，我们研究了运行在EC2云上的Apache Spark的编程灵活性、底层丰富性、调度能力和性能。通过利用数据局部性，Kira SE在Amazon EC2云上使用512个核心分析1TB数据集时，实现了2.5倍于等效C程序的加速。此外，通过利用最初为大数据基础设施设计的软件，Kira SE实现了与NERSC Edison超级计算机上运行的C实现竞争的性能。我们使用Kira的经验表明，Apache Spark等新兴大数据平台是多任务科学应用程序的高性能替代方案。
英文摘要：
Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, tools from the HPC software stack are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark -- a modern big data platform -- to parallelize many-task applications. We present Kira, a flexible and distributed astronomy image processing toolkit using Apache Spark. We then use the Kira toolkit to implement a Source Extractor application for astronomy images, called Kira SE. With Kira SE as the use case, we study the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the EC2 cloud. By exploiting data locality, Kira SE achieves a 2.5x speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, we show that by leveraging software originally designed for big data infrastructure, Kira SE achieves competitive performance to the C implementation running on the NERSC Edison supercomputer. Our experience with Kira indicates that emerging Big Data platforms such as Apache Spark are a performant alternative for many-task scientific applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
