Sampling multiple chain-of-thought rationales from a language model can uncover correct solutions missed by a single attempt. However, these rationales are typically lengthy, making repeated autoregressive generation costly. Continuous latent reasoning offers a compact alternative, but point-valued methods condition repeated answer sampling on a single rationale, which can limit exploration of diverse paths and answer coverage. We thus propose rationale distribution distillation (RDD), training a conditional flow over fixed-length latent sequences by distilling from multiple textual reasoning paths. Furthermore, we expand the distillation targets through within-query mixup, interpolating between complete latent rationales for the same query. The flow nonautoregressively generates the entire latent sequence, and a frozen language model autoregressively decodes answers from sampled rationale contexts. Empirically, the resulting distributional reasoner achieves higher answer coverage than the evaluated point-valued latent reasoning baselines, particularly on advanced mathematical tasks.
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Computer Science, University of Illinois at Chicago(芝加哥大学计算机科学系)
Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models
Hong Kong Baptist University(香港 Baptist 大学);Shanxi University(山西大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海先进研究院);Hong Kong University of Science and Technology(香港科技大学);The University of Manchester(曼彻斯特大学)
Tailoring Self-Rationalizers with Multi-Reward Distillation
University of Southern California(南加州大学);University of Washington(华盛顿大学);University of California Los Angeles(加州大学洛杉矶分校);Allen Institute for Artificial Intelligence(艾伦人工智能研究所)
Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究所);Nankai University(南开大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州));School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院)
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(中国科学院智能信息处理重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)