AutoFlow：用于分布式流处理的热点感知动态负载平衡
AutoFlow: Hotspot-Aware, Dynamic Load Balancing for Distributed Stream Processing
摘要：
流应用程序被广泛部署在云上。虽然Flink和Spark Streaming等现代分布式流媒体系统可以高效地调度和执行它们，但流媒体数据流通常动态变化，这可能会导致计算不平衡和反压。我们介绍AutoFlow，一个自动的，热点感知的动态负载平衡系统的流媒体流。它采用了一个集中式调度器，动态地监视整个子网中的负载平衡，并相应地实现状态迁移。调度器使用简单的异步分布式控制消息机制和热点减少算法来实现这两个任务。定时机制支持隐式屏障和高效的状态迁移，而无需全局屏障或暂停操作符。它还支持基于时间窗口的负载平衡测量，并将它们馈送到热点减少算法，而无需用户干扰。我们在Ray之上实现了AutoFlow，Ray是一个基于参与者的分布式执行框架。我们基于各种流基准数据集的评估表明，AutoFlow实现了良好的负载平衡，并在高度数据偏斜的工作负载下产生低延迟开销。
英文摘要：
Stream applications are widely deployed on the cloud. While modern distributed streaming systems like Flink and Spark Streaming can schedule and execute them efficiently, streaming dataflows are often dynamically changing, which may cause computation imbalance and backpressure. We introduce AutoFlow, an automatic, hotspot-aware dynamic load balance system for streaming dataflows. It incorporates a centralized scheduler which monitors the load balance in the entire dataflow dynamically and implements state migrations correspondingly. The scheduler achieves these two tasks using a simple asynchronous distributed control message mechanism and a hotspot-diminishing algorithm. The timing mechanism supports implicit barriers and a highly efficient state-migration without global barriers or pauses to operators. It also supports a time-window based load-balance measurement and feeds them to the hotspot-diminishing algorithm without user interference. We implemented AutoFlow on top of Ray, an actor-based distributed execution framework. Our evaluation based on various streaming benchmark dataset shows that AutoFlow achieves good load-balance and incurs a low latency overhead in highly data-skew workload.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
