一种快速、可扩展、通用的分布式数据聚合方法
A Fast, Scalable, Universal Approach For Distributed Data Aggregations
摘要:
在当前的大数据时代,数据工程已经成为许多科学分支的重要研究领域。人工智能(AI)的进步扩大了数据工程的范围,并在企业和研究界开辟了新的应用。聚合(在函数式编程中也称为reduce)是这些应用程序中不可或缺的功能。传统上,它们的目标是在大型数据集上生成有意义的信息,而今天,它们被用于为复杂的AI模型设计更有效的功能。聚合通常在数据抽象(如表/数组)之上执行,并与其他操作(如值分组)相结合。有些框架在上述领域中表现出色。但是,我们认为,数据分析工具必须能够与现有框架普遍集成,从而提高整个数据分析管道的生产力和效率。赛昂人努力填补这一空白。在本文中,我们提出了Cylon的快速和可扩展的聚合操作上实现的分布式内存表结构,普遍集成与现有的框架。
英文摘要:
In the current era of Big Data, data engineering has transformed into an essential field of study across many branches of science. Advancements in Artificial Intelligence (AI) have broadened the scope of data engineering and opened up new applications in both enterprise and research communities. Aggregations (also termed reduce in functional programming) are an integral functionality in these applications. They are traditionally aimed at generating meaningful information on large data-sets, and today, they are being used for engineering more effective features for complex AI models. Aggregations are usually carried out on top of data abstractions such as tables/ arrays and are combined with other operations such as grouping of values. There are frameworks that excel in the said domains individually. But, we believe that there is an essential requirement for a data analytics tool that can universally integrate with existing frameworks, and thereby increase the productivity and efficiency of the entire data analytics pipeline. Cylon endeavors to fulfill this void. In this paper, we present Cylon's fast and scalable aggregation operations implemented on top of a distributed in-memory table structure that universally integrates with existing frameworks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。