通过并发算法设计和新的抽象数据类型实现高效的数据流多路聚合
Efficient data streaming multiway aggregation through concurrent algorithmic designs and new abstract data types
摘要：
数据流依赖于连续查询以实时方式处理无限的数据流。由于相关应用程序涉及非常大量的数据，因此通常需要计算能力。数据结构充当接合点并维护数据流操作符的状态，潜在地支持高并行性并平衡它们之间的工作。基于这一事实，在这项工作中，我们研究和分析这些关节点的并行化需求，重点是流多路聚合的问题，其中从多个输入流接收大量数据。并行化需求的分析，以及现有的聚合设计及其数据结构的使用和局限性，使我们确定需要适当的共享对象，可以实现低延迟和高吞吐量的多路聚合。我们提出的要求，这些对象作为抽象的数据类型，我们提供了高效的无锁线性化算法实现，以及新的多路聚合算法设计，利用它们，支持确定性的顺序敏感和顺序不敏感的聚合函数。此外，我们指出了通过这些贡献打开的未来方向。本文包括一个广泛的实验研究，基于各种聚合连续查询两个大型数据集提取SoundCloud，音乐社交网络，并从智能电网网络。在所有的实验中，所提出的数据结构和增强的聚合运算符的处理性能显着提高，高达一个数量级，在吞吐量和延迟方面，在常用的技术基于队列。
英文摘要：
Data streaming relies on continuous queries to process unbounded streams of data in a real-time fashion. It is commonly demanding in computation capacity, given that the relevant applications involve very large volumes of data. Data structures act as articulation points and maintain the state of data streaming operators, potentially supporting high parallelism and balancing the work between them. Prompted by this fact, in this work we study and analyze parallelization needs of these articulation points, focusing on the problem of streaming multiway aggregation, where large data volumes are received from multiple input streams. The analysis of the parallelization needs, as well as of the use and limitations of existing aggregate designs and their data structures, leads us to identify needs for proper shared objects that can achieve low-latency and high throughput multiway aggregation. We present the requirements of such objects as abstract data types and we provide efficient lock-free linearizable algorithmic implementations of them, along with new multiway aggregate algorithmic designs that leverage them, supporting both deterministic order-sensitive and order-insensitive aggregate functions. Furthermore, we point out future directions that open through these contributions. The paper includes an extensive experimental study, based on a variety of aggregation continuous queries on two large datasets extracted from SoundCloud, a music social network, and from a Smart Grid network. In all the experiments, the proposed data structures and the enhanced aggregate operators improved the processing performance significantly, up to one order of magnitude, in terms of both throughput and latency, over the commonly-used techniques based on queues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
