消息代理系统中消费者组自动扩展的多目标优化
Multi-Objective Optimization of Consumer Group Autoscaling in Message Broker Systems
AI总结:
针对消息代理系统中消费者组自动扩展问题,提出多目标优化策略,同时最小化消费者数量与队列迁移,相比Kafka策略,在同等资源下将第90百分位延迟从217秒降至4.52秒。
摘要:
消息代理通常通过在有序分布式队列中添加可变大小的消息来协调数据生产者与消费者之间的通信。我们的目标是确定所需的消费者数量以及消费者-分区分配方案,以确保数据消费速率与数据生产速率保持同步。我们将该问题建模为可变物品尺寸的装箱问题。随着生产速率的变化,需要重新计算消费者-分区分配,这可能要求将分区从一个消费者重新平衡到另一个消费者。在重新平衡队列期间,正在写入该队列的数据不会被读取,从而导致额外的延迟成本。因此,我们关注多目标优化成本,即同时最小化消费者数量和队列迁移次数。我们提出了多种算法,并将它们与针对该应用场景的既有装箱启发式算法进行比较。将我们提出的消费者组分配策略与常用的Kafka策略进行对比,在两者使用相同消费者数量的情况下,我们的策略实现了第90百分位延迟为4.52秒,而Kafka为217秒。Kafka的分配策略仅在配置资源比我们的方法至少多60%时,才能在延迟方面改善消费者组的性能。
英文摘要:
Message brokers often mediate communication between data producers and consumers by adding variable-sized messages to ordered distributed queues. Our goal is to determine the number of consumers and consumer-partition assignments needed to ensure that the rate of data consumption keeps up with the rate of data production. We model the problem as a variable item size bin packing problem. As the rate of production varies, new consumer-partition assignments are computed, which may require rebalancing a partition from one consumer to another. While rebalancing a queue, the data being produced into the queue is not read leading to additional latency costs. As such, we focus on the multi-objective optimization cost of minimizing both the number of consumers and queue migrations. We present a variety of algorithms and compare them to established bin packing heuristics for this application. Comparing our proposed consumer group assignment strategy with Kafka's, a commonly employed strategy, our strategy presents a 90th percentile latency of 4.52s compared to Kafka's 217s with both using the same amount of consumers. Kafka's assignment strategy only improved the consumer group's performance with regards to latency with configurations that used at least 60% more resources than our approach.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。