Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of generating at least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome is achieved through a novel distillation method for discrete diffusion models, which reduces the number of inference steps by a factor of 32-64. Practically, at the 1.3B parameters scale, diffusion models, even without caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV-caching, and we anticipate further improvements with the inclusion of caching. Moreover, we demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.
Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification
ETH Zürich(苏黎世联邦理工学院);University of Copenhagen(哥本哈根大学);University of Science and Technology of China(中国科学技术大学);University of Amsterdam(阿姆斯特丹大学);East China Normal University(东华大学)
Quantification of Large Language Model Distillation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Peking University(北京大学);SUSTech(南方科技大学);SUAT;Leibowitz AI;UNSW Sydney(新南威尔士大学悉尼分校)
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学);AI Singapore(AI新加坡);National University of Singapore(新加坡国立大学);Institute for Infocomm Research (I$^2$R)(信息通信研究所);A$^{*}$STAR(A*星)
Department of Computer Science(计算机科学系);Engineering, Shanghai Jiao Tong University(上海交通大学工程学院);Shanghai Innovation Institute(上海创新研究院);Large Language Department, Tencent(腾讯大语言部门)
DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation
VILA Lab(VILA实验室);Mohamed bin Zayed University of AI(莫扎伊德·本·泽德人工智能大学);McGill University(麦吉尔大学);National University of Science and Technology(国家科学与技术大学)
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
University of Georgia(佐治亚大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Harvard University(哈佛大学);Carnegie Mellon University(卡内基梅隆大学);Vanderbilt University(范德堡大学);Mayo Clinic Arizona(亚利桑那梅奥诊所);Augusta University(奥古斯塔大学)
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Computer Science, University of Illinois at Chicago(芝加哥大学计算机科学系)
Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR))
A Survey on Knowledge Distillation of Large Language Models
University of Maryland(马里兰大学);Microsoft(微软公司);University of Technology Sydney(悉尼科技大学);Peking University(北京大学);The University of Sydney(悉尼大学);The University of Hong Kong(香港大学)