Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing (NLP) tasks. However, these models are often difficult to deploy due to significant computational requirements and resource constraints. Knowledge distillation (KD) is an effective technique for transferring the performance of larger LLMs to smaller models. Traditional KD methods primarily focus on the direct output of the teacher model, with little emphasis on the role of prompting during knowledge transfer. In this paper, we propose a set of novel response-priming prompting strategies applied in the knowledge distillation pipeline to enhance the performance of student models. Our approach fine-tunes a smaller Llama 3.1 8B Instruct model by distilling knowledge from a quantized Llama 3.1 405B Instruct teacher model. We apply LoRA optimization and evaluate on the GSM8K benchmark. Experimental results demonstrate that integrating reasoning-eliciting prompting into the proposed KD pipeline significantly improves student model performance, offering an efficient way to deploy powerful models in resource-constrained environments. We find that Ground Truth prompting results in a 55\% performance increase on GSM8K for a distilled Llama 3.1 8B Instruct compared to the same model distilled without prompting. A thorough investigation into the self-attention layers of the student models indicates that the more successful prompted models tend to exhibit certain positive behaviors inside their attention heads which can be tied to their increased accuracy. Our implementation can be found at https://github.com/alonso130r/knowledge-distillation.
Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
Fraunhofer Institute for Intelligent Analysis and Information Systems(弗劳恩霍夫智能分析与信息系统研究所);University of Southern Denmark(南丹麦大学);Fraunhofer Institute for Applied Information Technology(弗劳恩霍夫应用信息科技研究所);University Hospital of Cologne(科隆大学医院)
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
School of Foreign Studies, Hefei University of Technology(合肥工业大学外国语学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Hefei Institute of Technology(合肥学院)
SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation
The University of Liverpool(利物浦大学);University of Macau(澳门大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Microsoft(微软);Shanghai University of International Business and Economics(上海国际经贸管理大学);Huizhou University(惠州市大学)
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models
KAIST(韩国科学技术院);Theori;Université de Montréal(蒙特利尔大学);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Technische Hochschule Nürnberg Georg Simon Ohm(纽伦堡应用技术大学);CIFAR AI Chair(加拿大高级研究所人工智能主席项目)
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
University of California Los Angeles(加利福尼亚大学洛杉矶分校);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Southern California(南加利福尼亚大学);Yonsei University(延世大学);University of Washington(华盛顿大学)
Boosting Large Language Models for Mental Manipulation Detection via Data Augmentation and Distillation
Zhejiang University(浙江大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信)
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Institute of Digital Twin, EIT(数字孪生研究院);Saarland University(萨尔兰大学);Meituan Inc.(美团公司)
Reasoning Beyond Limits: Advances and Open Problems for LLMs
United Arab Emirates University(阿拉伯联合酋长国大学);Technology Innovation Institute(技术创新研究所);Eötvös Loránd University(厄特沃什·洛朗大学);Khalifa University of Science(谢赫扎耶德科学技术大学)