Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.
DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
VILA Lab(VILA实验室);Mohamed bin Zayed University of AI(莫扎伊德·本·泽德人工智能大学);McGill University(麦吉尔大学);National University of Science and Technology(国家科学与技术大学)
Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models
Zhejiang University(浙江大学);School of Public Health, Zhejiang University(浙江大学公共卫生学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Liangzhu Laboratory and Institute of Wenzhou, Zhejiang University(浙江大学良渚实验室与温州研究院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Huawei Inc.(华为公司)
Large Language Model Distilling Medication Recommendation Model
Xi'an Jiaotong University(西安交通大学);City University of Hong Kong(香港城市大学);Jarvis Research Center, Tencent YouTu Lab(腾讯优图实验室贾维斯研究中心);Southern University of Science and Technology(南方科技大学);Jilin University(吉林大学)
Distilling Rule-based Knowledge into Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Pattern Recognition Center, WeChat AI, Tencent Inc.(腾讯微信AI模式识别中心)
Distilling Closed-Source LLM's Knowledge for Locally Stable and Economic Biomedical Entity Linking
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);Changsha Aier Eye Hospital(长沙爱尔眼科医院);Portland State University(波特兰州立大学)
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Institute of Digital Twin, EIT(数字孪生研究院);Saarland University(萨尔兰大学);Meituan Inc.(美团公司)
Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Baidu Inc(百度公司)
Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation
School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)
Distilling Lightweight Language Models for C/C++ Vulnerabilities
School of Computer Science and Technology(计算机科学与技术学院);Beijing Institute of Technology(北京理工大学);School of Cyberspace Science and Technology(网络空间科学与技术学院);School of Computer Science(计算机科学学院);University of Auckland(奥克兰大学)
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of Chicago(芝加哥大学);The Chinese University of Hong Kong(香港中文大学);East China Normal University(华东师范大学)
SLMRec: Distilling Large Language Models into Small for Sequential Recommendation
Rutgers University(新泽西罗格斯大学);Broad Institute of MIT and Harvard(哈佛大学与麻省理工学院Broad研究所);Ant Group(蚂蚁集团);Dian Diagnostics Group Co.(迪安诊断集团);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Technology Sydney(悉尼技术大学)