交付、一致性和确定性:重新思考分布式流处理中的保证
Delivery, consistency, and determinism: rethinking guarantees in distributed stream processing
摘要:
最先进的流处理系统的一致性要求是根据交付保证来定义的。精确一次是最强的一个,也是最终用户最想要的。然而,关于这一概念有几个问题。强制执行一次的常用技术会产生显著的性能开销。此外,精确一次的概念没有正式定义,也没有捕获提供支持此保证的流处理系统的所有属性。在本文中,我们介绍了一个正式的框架,使我们能够更经常地定义流保证。我们证明了交付,一致性和确定性的属性是紧密相连的分布式流处理。我们还表明,具有轻量级的确定性,它是可能的,以提供确切的一次几乎没有性能开销。实验表明,该方法可以显着优于替代工业解决方案。
英文摘要:
Consistency requirements for state-of-the-art stream processing systems are defined in terms of delivery guarantees. Exactly-once is the strongest one and the most desirable for end-user. However, there are several issues regarding this concept. Commonly used techniques that enforce exactly-once produce significant performance overhead. Besides, the notion of exactly-once is not formally defined and does not capture all properties that provide stream processing systems supporting this guarantee. In this paper, we introduce a formal framework that allows us to define streaming guarantees more regularly. We demonstrate that the properties of delivery, consistency, and determinism are tightly connected within distributed stream processing. We also show that having lightweight determinism, it is possible to provide exactly-once with almost no performance overhead. Experiments show that the proposed approach can significantly outperform alternative industrial solutions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。