大数据处理程序的抽象视图
An Abstract View of Big Data Processing Programs
摘要：
提出了一种基于数据流的并行数据处理程序的模型，该模型与目标大数据处理框架无关。本文重点研究了非迭代和迭代程序的形式化抽象规范，概括了数据流大数据处理框架所采用的策略。所提出的模型依赖于Monoid AlgebraandPetri Netto在两个级别上抽象大数据处理程序：表示程序数据流的高级别和表示数据转换操作的低级别（例如，过滤、聚合、连接）。我们扩展了[1]中提出的数据处理程序的模型，使迭代程序的使用成为可能。考虑到迭代和贪婪大数据分析算法的民主化，由基于数据流的并行编程模型实现的迭代数据处理程序的通用规范是必不可少的。事实上，这些算法要求重新访问并行编程模型来表达迭代。本文对Apache Spark、DryadLINQ、Apache Beam和Apache Flink提出的迭代策略进行了比较分析。讨论了该模型如何实现对这些策略的推广。
英文摘要：
This paper proposes a model for specifying data flow based parallel data processing programs agnostic of target Big Data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow Big Data processing frameworks. The proposed model relies on monoid AlgebraandPetri Netstoabstract Big Data processing programs in two levels: a high level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1], to enable the use of iterative programs. The general specification of iterative data processing programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy Big Data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed byApache Spark, DryadLINQ, Apache Beam and Apache Flink. It discusses how the model achieves to generalize these strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
