BlobShuffle：通过对象存储实现流处理系统中经济高效的重分区——以Kafka Streams为例
BlobShuffle: Cost-Effective Repartitioning in Stream Processing Systems via Object Storage Exemplified with Kafka Streams
AI总结：
提出BlobShuffle方法，利用云对象存储作为中间交换层，通过批量存储和紧凑通知机制降低流处理系统中跨可用区数据重排的网络成本，实验表明相比原生Kafka Streams可减少40倍以上成本且延迟低于2秒。
摘要：
数据流的洗牌或重分区是现代流处理框架在分布式大规模环境下支持有状态工作负载的关键操作。然而，在当今的云部署中，由于跨多个可用区（AZ）的大量网络流量以及在高吞吐、强一致性消息骨干网大规模运行时的运维负担，洗牌可能成为主要的成本驱动因素。我们提出BlobShuffle，一种新颖的流处理系统经济高效洗牌方法，它利用云对象存储作为中间交换层。BlobShuffle不是直接发送所有洗牌记录，而是将记录分组为批次，将这些批次存储在云对象存储中，并仅转发紧凑的通知。下游操作员使用这些通知检索相关批次并提取相应记录。BlobShuffle通过可配置的批处理和分布式缓存机制平衡成本效率和延迟。BlobShuffle作为Kafka Streams的附加组件实现，仅需对现有应用程序进行最少的代码更改，不修改Kafka和底层基础设施，并保留Kafka Streams的一致性和正确性保证。在基于Kubernetes的AWS部署上进行的大规模实验评估中，我们表明BlobShuffle可以将洗牌成本降低到原生Kafka Streams洗牌的40倍以上，同时将95百分位洗牌延迟保持在2秒以下。此外，它能够扩展到处理超过2 GiB/s的数据，而在我们的实验中未遇到可扩展性限制，表明BlobShuffle可以在大规模下经济地支持洗牌密集型工作负载。
英文摘要：
Shuffling or repartitioning data streams is an essential operation of state-of-the-art stream processing frameworks to support stateful workloads in a large-scale, distributed setting. In today's cloud deployments, however, shuffling can become a major cost driver due to substantial network traffic across multiple availability zones (AZs) as well as an operational burden when operating a high-throughput, strongly consistent messaging backbone at scale. We present BlobShuffle, a novel approach to cost-effective shuffling for stream processing systems that leverages cloud object storage as an intermediate exchange layer. Instead of sending all shuffled records directly, BlobShuffle groups records into batches, stores these batches in cloud object storage, and forwards only compact notifications. Downstream operators use these notifications to retrieve the relevant batches and extract the corresponding records. BlobShuffle balances cost efficiency and latency through configurable batching and a distributed caching mechanism. BlobShuffle is implemented as an add-on for Kafka Streams that requires only minimal code changes to existing applications, leaves Kafka and the underlying infrastructure unmodified, and preserves Kafka Streams' consistency and correctness guarantees. In a large-scale experimental evaluation on a Kubernetes-based AWS deployment, we show that BlobShuffle can reduce shuffling costs by more than 40x compared to native Kafka Streams shuffling while keeping the 95th percentile shuffle latency below 2 seconds. Moreover, it scales to processing more than 2 GiB/s without encountering a scalability limit in our experiments, indicating that BlobShuffle can economically support shuffle-intensive workloads at large scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
