Large Language Models (LLMs) face significant challenges at inference time due to their high computational demands. To address this, we present Performance-Guided Knowledge Distillation (PGKD), a cost-effective and high-throughput solution for production text classification applications. PGKD utilizes teacher-student Knowledge Distillation to distill the knowledge of LLMs into smaller, task-specific models. PGKD establishes an active learning routine between the student model and the LLM; the LLM continuously generates new training data leveraging hard-negative mining, student model validation performance, and early-stopping protocols to inform the data generation. By employing a cyclical, performance-aware approach tailored for highly multi-class, sparsely annotated datasets prevalent in industrial text classification, PGKD effectively addresses training challenges and outperforms traditional BERT-base models and other knowledge distillation methods on several multi-class classification datasets. Additionally, cost and latency benchmarking reveals that models fine-tuned with PGKD are up to 130X faster and 25X less expensive than LLMs for inference on the same classification task. While PGKD is showcased for text classification tasks, its versatile framework can be extended to any LLM distillation task, including language generation, making it a powerful tool for optimizing performance across a wide range of AI applications.
Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Information Technology and Management, University of International Business and Economics(对外经济贸易大学信息与技术管理学院)
LLM-Guided Knowledge Distillation for Temporal Knowledge Graph Reasoning
School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);School of Computing and Artificial Intelligence, Southwest Jiaotong University(西南交通大学计算机与人工智能学院);School of Information Science and Engineering, Chongqing Jiaotong University(重庆交通大学信息科学与工程学院)
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
School of Foreign Studies, Hefei University of Technology(合肥工业大学外国语学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Hefei Institute of Technology(合肥学院)
Knowledge Hierarchy Guided Biological-Medical Dataset Distillation for Domain LLM Training
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences(中国科学院大学杭州高等研究院)