Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosting the capabilities of smaller models through knowledge distillation (KD) from these powerful yet black-box teachers. While leveraging the high-quality outputs of these teachers is advantageous, the inaccessibility of their internal states often limits effective knowledge transfer. To overcome this limitation, we introduce Proxy-KD, a novel method that uses a proxy model to facilitate the efficient transfer of knowledge from black-box LLMs to smaller models. Our experiments show that Proxy-KD not only enhances the performance of KD from black-box teacher models but also surpasses traditional white-box KD techniques.~This approach presents a compelling new avenue for distilling knowledge from advanced LLMs.
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院);School of Future Technology, Tianjin University, China(天津大学未来技术学院);Georgia Tech Shenzhen Institute, Tianjin University, China(Georgia Tech深圳研究院);School of Artificial Intelligence, Tianjin University, China(天津大学人工智能学院)
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Tsinghua University(清华大学);Zhejiang University(浙江大学);Peking University(北京大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Ten Words Only Still Help: Improving Black-Box AI-Generated Text Detection via Proxy-Guided Efficient Re-Sampling
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(中国科学院智能信息处理重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Xi’an Jiaotong University(西安交通大学)
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州));Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);Renmin University of China(中国人民大学);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学)