交互式分析十亿对象目录：Apache Spark for physicians
Analyzing billion-objects catalog interactively: Apache Spark for physicists
摘要：
Apache Spark是一个大数据框架，用于处理大型分布式数据集。虽然它在工业中被广泛使用，但在学术界仍然相当有限，或者通常仅限于软件工程师。本文的目标是通过实际用例表明，该技术已经足够成熟，可以在天文学家或宇宙学家没有过多编程技能的情况下使用，以便对大型数据集进行标准分析，例如来自未来星系调查的数据集。为了证明这一点，我们从对应于10年LSST数据（60亿个星系）的现实模拟开始。然后，我们设计，优化和基准测试一组Spark Python算法，以执行标准操作，如添加光度红移误差，测量选择函数或计算功率谱超过层析箱。大多数命令在数十秒内对完整的110 GB数据集执行，因此可以交互式执行，以设计全面的宇宙学分析。jupyter笔记本总结了分析结果，可在https://github.com/astrolabsoftware/1807.03078上查阅。
英文摘要：
Apache Spark is a Big Data framework for working on large distributed datasets. Although widely used in the industry, it remains rather limited in the academic community or often restricted to software engineers. The goal of this paper is to show with practical uses-cases that the technology is mature enough to be used without excessive programming skills by astronomers or cosmologists in order to perform standard analyses over large datasets, as those originating from future galaxy surveys. To demonstrate it, we start from a realistic simulation corresponding to 10 years of LSST data taking (6 billions of galaxies). Then, we design, optimize and benchmark a set of Spark python algorithms in order to perform standard operations as adding photometric redshift errors, measuring the selection function or computing power spectra over tomographic bins. Most of the commands execute on the full 110 GB dataset within tens of seconds and can therefore be performed interactively in order to design full-scale cosmological analyses. A jupyter notebook summarizing the analysis is available at https://github.com/astrolabsoftware/1807.03078.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
