FuxiShuffle：一种适应性强且具有容错性的 shuffle 服务，用于阿里云分布式数据处理
FuxiShuffle: An Adaptive and Resilient Shuffle Service for Distributed Data Processing on Alibaba Cloud
AI总结：
FuxiShuffle 是一种为阿里云 MaxCompute 超大规模生产环境设计的 shuffle 服务，通过动态模式选择、进度感知调度和多副本故障转移等技术，提升适应性和故障恢复效率。
摘要：
在分布式数据处理中，shuffle 用于在上游和下游操作符之间交换中间结果，通常是瓶颈，因为诸如小随机 I/O 和网络竞争等因素。已经设计了几种系统来提高 shuffle 效率，但根据我们在阿里云 MaxCompute 平台运行超大规模集群的经验，我们发现它们无法适应高度动态的任务特征和集群资源条件，且其容错机制在故障不可避免时是被动且低效的。为解决这些限制，我们设计并实现了 FuxiShuffle 作为 MaxCompute 超大规模生产环境中的通用数据 shuffle 服务，具有良好的适应性和高效的故障恢复能力。具体来说，为了实现良好的适应性，FuxiShuffle 根据运行时信息动态选择 shuffle 模式，对下游工作者进行进度感知调度，并自动确定每个 shuffle 数据分片最适合的备份策略。为了使故障恢复高效，FuxiShuffle 通过多副本故障转移主动确保数据可用性，通过谨慎的内存管理防止内存溢出，并采用一种增量恢复机制，该机制不会丢失计算进度。我们的实验表明，与基准系统相比，FuxiShuffle 不仅显著减少了端到端作业完成时间，还减少了聚合资源消耗。微实验表明，我们的设计在提高适应性和故障恢复能力方面是有效的。
英文摘要：
Shuffle exchanges intermediate results between upstream and downstream operators in distributed data processing and is usually the bottleneck due to factors such as small random I/Os and network contention. Several systems have been designed to improve shuffle efficiency, but from our experiences of running ultra-large clusters at Alibaba Cloud MaxCompute platform, we observe that they can not adapt to highly dynamic job characteristics and cluster resource conditions, and their fault tolerance mechanisms are passive and inefficient when failures are inevitable. To tackle their limitations, we design and implement FuxiShuffle as a general data shuffle service for the ultra-large production environment of MaxCompute, featuring good adaptability and efficient failure resilience. Specifically, to achieve good adaptability, FuxiShuffle dynamically selects the shuffle mode based on runtime information, conducts progress-aware scheduling for the downstream workers, and automatically determines the most suitable backup strategy for each shuffle data chunk. To make failure resilience efficient, FuxiShuffle actively ensures data availability with multi-replica failover, prevents memory overflow with careful memory management, and employs an incremental recovery mechanism that does not lose computation progress. Our experiments show that, compared to baseline systems, FuxiShuffle significantly reduces not only end-to-end job completion time but also aggregate resource consumption. Micro experiments suggest that our designs are effective in improving adaptability and failure resilience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
