arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.15992cs.LGcs.AI
收藏

Stratos:分布式云环境下面向定制化 LLM 的端到端蒸馏流水线

Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments

Ziming Dai, Tuo Zhang, Fei Gao, Xingyi Cai, Xiaofei Wang, Cheng Zhang, Wenyu Wang, Chengjie Zang

AI总结:

本文提出 Stratos,一种在分布式云环境中自动化服务器与模型选择、知识蒸馏及部署的端到端 LLM 流水线,可在预算与性能约束下降低延迟和成本,并在麻将推理任务上取得优于 GPT-4o 教师基线的效果。

摘要:

垂直、特定领域任务的兴起,以及在延迟和预算等约束下优化性能的需求,推动了工业界对定制化且具备成本效益的大型语言模型(LLM)日益增长的需求。知识蒸馏作为一种高效的模型压缩与迁移技术,提供了可行的解决方案。然而,现有蒸馏框架通常需要人工干预,并且难以满足此类复杂的用户自定义蒸馏需求。为弥补这一差距,我们提出 Stratos,这是一种端到端的 LLM 蒸馏流水线,可在分布式云环境中自动化完成服务器与模型选择、知识蒸馏以及部署。在给定用户定义的模型性能和系统预算约束的情况下,Stratos 会自动选择 Pareto 最优服务器,动态匹配教师—学生对,并根据任务复杂度调整蒸馏策略,以优化云托管。实验表明,在一项罕见的、特定领域的麻将推理任务上,借助逆向合成数据和知识注入,Stratos 生成的学生模型达到了其 GPT-4o 教师基线四倍的准确率。此外,它在不损失准确率的情况下降低了延迟和成本。这些结果凸显了其在垂直领域 LLM 部署方面的前景。

英文摘要:

The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑