Apache Spark的FITS数据源
FITS Data Source for Apache Spark
摘要：
我们调查Apache Spark的性能，集群计算框架，用于分析未来LSST类星系调查的数据。Apache Spark解决大数据问题的尝试迄今为止在业界证明是成功的，但它在天文学界的应用仍然有限。我们将展示如何管理复杂的二进制数据结构处理天体物理学实验，如二进制表存储在FITS文件中，在分布式环境中。为此，我们首先设计并实现了一个Spark连接器来处理任意大的FITS文件集，称为spark-fits。用户界面是这样的，一个简单的文件“拖放”到一个集群，充分利用了框架。我们使用LSST快速仿真工具CoLoRe演示了火花拟合的高度可扩展性，并提出了测量和调整工作负载性能瓶颈的方法，在位于巴黎南大学的Cloud@VirtualData上扩展到TB的FITS数据。我们还评估了它的性能Cori，高性能计算系统位于NERSC，并广泛用于科学界。
英文摘要：
We investigate the performance of Apache Spark, a cluster computing framework, for analyzing data from future LSST-like galaxy surveys. Apache Spark attempts to address big data problems have hitherto proved successful in the industry, but its use in the astronomical community still remains limited. We show how to manage complex binary data structures handled in astrophysics experiments such as binary tables stored in FITS files, within a distributed environment. To this purpose, we first designed and implemented a Spark connector to handle sets of arbitrarily large FITS files, called spark-fits. The user interface is such that a simple file "drag-and-drop" to a cluster gives full advantage of the framework. We demonstrate the very high scalability of spark-fits using the LSST fast simulation tool, CoLoRe, and present the methodologies for measuring and tuning the performance bottlenecks for the workloads, scaling up to terabytes of FITS data on the Cloud@VirtualData, located at Université Paris Sud. We also evaluate its performance on Cori, a High-Performance Computing system located at NERSC, and widely used in the scientific community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
