Daedalus：面向分布式流处理系统资源效率的自适应水平自动扩展
Daedalus: Self-Adaptive Horizontal Autoscaling for Resource Efficiency of Distributed Stream Processing Systems
AI总结：
Daedalus通过自适应机制实现分布式流处理系统的高效水平自动扩展，减少资源消耗并保持低延迟性能。
摘要：
分布式流处理（DSP）系统能够处理大量无界数据流，提供高吞吐量和低延迟。为了保持稳定的服务质量（QoS），这些系统需要足够的资源分配。同时，过度配置会导致能源浪费和高运营成本。因此，为最大化资源利用率，提出了自动扩展方法，旨在高效匹配资源分配与输入工作负载。然而，确定何时以及如何扩展仍是一个重大挑战。鉴于DSP任务的长时间运行性质，扩展操作需要在运行时执行，并且为了保持良好的QoS，它们应既准确又不频繁。为了解决自动扩展的挑战，自适应系统概念尤为合适。这些系统监控自身和环境，适应变化，几乎不需要专家参与。本文介绍了Daedalus，一种用于DSP系统自动扩展的自适应管理者，它借鉴自适应原则来解决高效自动扩展的挑战。Daedalus监控正在运行的DSP任务并构建性能模型，旨在预测不同扩展规模下的最大处理能力。当结合时间序列预测来预测未来的工作负载时，Daedalus会主动扩展DSP任务，优化最大吞吐量并最小化延迟和资源使用。我们使用Apache Flink和Kafka Streams进行了实验，评估Daedalus与两种最先进的方法的性能。Daedalus能够实现相同比例的延迟，同时将资源使用量减少高达71%。
英文摘要：
Distributed Stream Processing (DSP) systems are capable of processing large streams of unbounded data, offering high throughput and low latencies. To maintain a stable Quality of Service (QoS), these systems require a sufficient allocation of resources. At the same time, over-provisioning can result in wasted energy and high operating costs. Therefore, to maximize resource utilization, autoscaling methods have been proposed that aim to efficiently match the resource allocation with the incoming workload. However, determining when and by how much to scale remains a significant challenge. Given the long-running nature of DSP jobs, scaling actions need to be executed at runtime, and to maintain a good QoS, they should be both accurate and infrequent. To address the challenges of autoscaling, the concept of self-adaptive systems is particularly fitting. These systems monitor themselves and their environment, adapting to changes with minimal need for expert involvement. This paper introduces Daedalus, a self-adaptive manager for autoscaling in DSP systems, which draws on the principles of self-adaption to address the challenge of efficient autoscaling. Daedalus monitors a running DSP job and builds performance models, aiming to predict the maximum processing capacity at different scale-outs. When combined with time series forecasting to predict future workloads, Daedalus proactively scales DSP jobs, optimizing for maximum throughput and minimizing both latencies and resource usage. We conducted experiments using Apache Flink and Kafka Streams to evaluate the performance of Daedalus against two state-of-the-art approaches. Daedalus was able to achieve comparable latencies while reducing resource usage by up to 71%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
