NavP:通过应用程序启动的检查点启用用于科学数据处理的导航编程
NavP: Enabling Navigational Programming for Science Data Processing via Application-Initiated Checkpointing
摘要:
科学数据系统(SDS)处理从采集到处理再到分发的科学数据。它们现在部署在云中,云实例利用率的效率对于成功至关重要。传统的SDS无法利用具有成本效益的Amazon EC2现货市场,特别是对于长期运行的任务。在NASA/JPL目前的实践中发现的一些困难是:缺乏应用程序程序员保存有价值的部分结果以供未来处理继续的机制,使用基于容器(奇点)的沙箱超过200,000个操作系统级文件的沉重负担;以及在笔记本电脑上开发算法/程序的科学家和在云计算或超级计算中部署软件的SDS专家之间的差距。我们提出了第一个证明的原则,这使用NavP(导航编程)和容错计算(FTC)在SDS中,采用程序状态迁移促进检查点重启(C/R)。NavP为应用程序员提供了一个新的分布式计算导航视图。我们开发的DHP(DMTCP Hop and Publish)工具使应用程序员能够通过在应用程序代码中插入hop(destination)语句来导航实例或节点之间的计算,并选择何时发布他们认为值得将来继续的算法阶段的部分结果。使用DHP的结果是,并行分布式SDS变得更容易编程和部署,从而可以更有效地利用Amazon EC2 Spot市场。本技术报告描述了一个高层次的设计和初步实施。
英文摘要:
Science Data Systems (SDS) handle science data from acquisition through processing to distribution. They are deployed in the Cloud today, and the efficiency of Cloud instance utilization is critical to success. Conventional SDS are unable to take advantage of a cost-effective Amazon EC2 spot market, especially for long-running tasks. Some of the difficulties found in current practice at NASA/JPL are: a lack of mechanism for app programmers to save valuable partial results for future processing continuation, the heavy weight from using container-based (Singularity) sandboxes with more than 200,000 OS-level files; and the gap between scientists developing algorithms/programs on a laptop and the SDS experts deploying software in Cloud computing or supercomputing. We present a first proof-of-principle of this using NavP (Navigational Programming) and fault-tolerant computing (FTC) in SDS, by employing program state migration facilitated by Checkpoint-Restart (C/R). NavP provides a new navigational view of computations in a distributed world for the application programmers. The tool of DHP (DMTCP Hop and Publish) we developed enables the application programmers to navigate the computation among instances or nodes by inserting hop(destination) statements in their app code, and choose when to publish partial results at stages of their algorithms that they think worthwhile for future continuation. The result of using DHP is that a parallel distributed SDS becomes easier to program and deploy, and this enables more efficient leveraging of the Amazon EC2 Spot market. This technical report describes a high-level design and an initial implementation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。