arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.10169cs.DCcs.DB
收藏

Hazelcast Jet:在第99.99百分位的低延迟流处理

Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile

Can Gencer, Marko Topolnik, Viliam Ďurina, Emin Demirci, Ensar B. Kahveci, Ali Gürbüz Ondřej Lukáš, József Bartók, Grzegorz Gierlach, František Hartman, Ufuk Yı… 展开作者

Can Gencer, Marko Topolnik, Viliam Ďurina, Emin Demirci, Ensar B. Kahveci, Ali Gürbüz Ondřej Lukáš, József Bartók, Grzegorz Gierlach, František Hartman, Ufuk Yılmaz, Mehmet Doğan, Mohamed Mandouh, Marios Fragkoulis, Asterios Katsifodimos

摘要:

Jet是Hazelcast在过去五年中开发的一款开源、高性能、分布式流处理器。Jet的设计以毫秒级延迟为第99.99百分位数,作为其主要设计目标。最初,Jet的目标是成为一个执行引擎,在Hazelcast的内存数据网格(IMDG)生成的流之上执行复杂的业务逻辑:一组高性能、内存中、分区和复制的数据结构。随着时间的推移,Jet发展成为一个成熟的横向扩展流处理器,可以处理无序流和一次处理保证。Jet的端到端延迟在毫秒级,其吞吐量在每个CPU核心数百万个事件的数量级。本文介绍了我们为了最大限度地提高每个CPU核心的性能而做出的主要设计决策,以及经验教训和经验性能评估。

英文摘要:

Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet's purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast's In-memory Data Grid (IMDG): a set of high-performance, in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle out-of-order streams and exactly-once processing guarantees. Jet's end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑