NebulOS:天体物理学大数据框架
NebulOS: A Big Data Framework for Astrophysics
摘要:
我们介绍NebulOS,这是一个大数据平台,允许将Linux机器集群视为一台计算机。使用NebulOS,为数据中心编写大规模并行程序的过程并不比为台式计算机编写Python脚本复杂。该平台使大多数预先存在的数据分析软件能够在数据中心中作为规模使用而无需修改。浅的学习曲线和与现有软件的兼容性大大减少了开发分布式数据分析管道所需的时间。该平台基于行业标准的开源大数据技术构建,并继承了多项容错功能。NebulOS通过添加直观的用户界面、自动任务监控和其他可用性功能来增强这些技术。我们提出了一个架构的摘要,提供使用示例,并讨论了系统的性能扩展。
英文摘要:
We introduce NebulOS, a Big Data platform that allows a cluster of Linux machines to be treated as a single computer. With NebulOS, the process of writing a massively parallel program for a datacenter is no more complicated than writing a Python script for a desktop computer. The platform enables most pre-existing data analysis software to be used, as scale, in a datacenter without modification. The shallow learning curve and compatibility with existing software greatly reduces the time required to develop distributed data analysis pipelines. The platform is built upon industry-standard, open-source Big Data technologies, from which it inherits several fault tolerance features. NebulOS enhances these technologies by adding an intuitive user interface, automated task monitoring, and other usability features. We present a summary of the architecture, provide usage examples, and discuss the system's performance scaling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。