AntBatchInfer:Kubernetes集群中的弹性批量推理
AntBatchInfer: Elastic Batch Inference in the Kubernetes Cluster
AI总结:
针对工业离线批量推理的稳定性与效率难题,提出专为非专用Kubernetes集群优化的AntBatchInfer框架,通过多级容错、多级扩缩容等技术提升性能,实验优于基线2-6倍,已在蚂蚁集团大规模落地。
摘要:
离线批量推理是工业界深度学习应用中的常见任务,但在处理海量数据与复杂推理流水线时,保障稳定性与性能颇具挑战。本文提出了AntBatchInfer这一专为非专用集群优化的弹性批量推理框架。该框架通过多级容错能力应对上述挑战,可稳定执行各类长期运行的推理任务;还通过流水线、节点内及节点间扩缩容提升推理效率,并进一步优化了复杂多模型批量推理场景下的性能。通过大量实验与真实业务统计数据,我们验证了该框架在稳定性与效率方面的优势:实验中,其在单模型与多模型批量推理任务上的表现分别比基线方法至少高出2倍与6倍。目前该框架已在蚂蚁集团广泛应用,每日承载来自DLRM、计算机视觉(CV)、自然语言处理(NLP)等不同场景的数千个作业,证实了其工业实用性。
英文摘要:
Offline batch inference is a common task in the industry for deep learning applications, but it can be challenging to ensure stability and performance when dealing with large amounts of data and complicated inference pipelines. This paper demonstrated AntBatchInfer, an elastic batch inference framework, which is specially optimized for the non-dedicated cluster. AntBatchInfer addresses these challenges by providing multi-level fault-tolerant capabilities, enabling the stable execution of versatile and long-running inference tasks. It also improves inference efficiency by pipelining, intra-node, and inter-node scaling. It further optimizes the performance in complicated multiple-model batch inference scenarios. Through extensive experiments and real-world statistics, we demonstrate the superiority of our framework in terms of stability and efficiency. In the experiment, it outperforms the baseline by at least $2\times$ and $6\times$ in the single-model or multiple-model batch inference. Also, it is widely used at Ant Group, with thousands of daily jobs from various scenarios, including DLRM, CV, and NLP, which proves its practicability in the industry.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。