Spark-MPI：认知应用的第五范式
Spark-MPI: Approaching the Fifth Paradigm of Cognitive Applications
摘要：
在过去的十年中，数据密集型科学的第四范式迅速成为多个应用领域的主要驱动概念，包括并产生大型设备，如光源和尖端望远镜。数据密集型项目的成功随后引发了下一代机器学习方法。这些新的人工智能系统清楚地代表了从数据处理管道到复合认知应用的第五范式的范式转变，需要集成大数据处理平台和HPC技术。针对数据密集型生态系统和计算密集型生态系统之间存在的阻抗不匹配问题，提出了基于MPI Exascale Process Management Interface（PMIx）的Spark-MPI方法。该方法在混合MPI/GPU重叠关联图像重建管道和分布式深度学习应用程序的背景下进行了演示。
英文摘要：
Over the past decade, the fourth paradigm of data-intensive science rapidly became a major driving concept of multiple application domains encompassing and generating large-scale devices such as light sources and cutting edge telescopes. The success of data-intensive projects subsequently triggered the next generation of machine learning approaches. These new artificial intelligent systems clearly represent a paradigm shift from data processing pipelines towards the fifth paradigm of composite cognitive applications requiring the integration of Big Data processing platforms and HPC technologies. The paper addresses the existing impedance mismatch between data-intensive and compute-intensive ecosystems by presenting the Spark-MPI approach based on the MPI Exascale Process Management Interface (PMIx). The approach is demonstrated within the context of hybrid MPI/GPU ptychographic image reconstruction pipelines and distributed deep learning applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
