The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment.
CCS: Continuous Learning for Customized Incremental Wireless Sensing Services
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Huawei Technologies Co. Ltd(华为技术有限公司)
BearingPGA-Net: A Lightweight and Deployable Bearing Fault Diagnosis Network via Decoupled Knowledge Distillation and FPGA Acceleration
Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong Polytechnic University(香港理工大学);Analog Devices, Inc.(亚德诺公司);The Chinese University of Hong Kong(香港中文大学)
Distilling Desired Comments for Enhanced Code Review with Large Language Models
Software Institute Nanjing University(南京大学软件学院);Faculty of Science and Engineering Southern Cross University(南十字星大学科学与工程学院);Huawei Technologies Co., Ltd.(华为技术有限公司)
Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Science and Technology of China(中国科学技术大学);School of Intelligence Science and Technology, University of Science and Technology Beijing(北京科技大学智能科学与技术学院)