"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学);Nankai University(南开大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州));School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(软件学院,人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学)
Chain-of-Thought (CoT) reasoning empowers Large Language Models (LLMs) with remarkable capabilities but typically requires prohibitive parameter scales. CoT distillation has emerged as a promising paradigm to transfer reasoning prowess into compact Student Models (SLMs), but existing approaches often rely on a solitary teacher, capping the student's potential since individual LLMs often exhibit distinct capability biases and may suffer from catastrophic forgetting. While leveraging diverse teachers seems appealing, effectively fusing their supervisions remains challenging: teacher-student incompatibility risks amplifying hallucinations, and passive supervision fails to ensure genuine logic internalization. To address this, we introduce COMPACT, a framework that adaptively fuses supervisions from different teachers by dynamically weighting teacher gradients based on the student's real-time compatibility evaluated by a multi-dimensional metric: (1) Graph-based Consensus to filter misleading rationales by identifying mainstream reasoning paths; (2) Mutual-Information-based Adaptability to detect "epiphany moments" for genuinely understanding the reasoning process rather than merely imitating; and (3) Loss-based Difficulty to assess student receptivity to the teacher's guidance and prevent negative transfer. Extensive experiments and latent space analysis demonstrate that COMPACT effectively integrates diverse reasoning capabilities without damaging the model's original knowledge structure, achieving state-of-the-art performance on various benchmarks while mitigating catastrophic forgetting.
MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
South China Agricultural University(华南农业大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Electronic Science and Technology of China(电子科技大学)
Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学人工智能研究院计算机科学与技术系);School of Computer Science, Beijing University of Posts and Telecommunications, China(北京邮电大学计算机科学学院)
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Institute of Digital Twin, EIT(数字孪生研究院);Saarland University(萨尔兰大学);Meituan Inc.(美团公司)
Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University(吉林大学符号计算与知识工程重点实验室);Tongyi Lab, Alibaba Group(阿里集团通义实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SIGS, Tsinghua University(清华大学信息科学与技术学院);Southern University of Science and Technology(南方科技大学);Guangdong Laboratory of AI and Digital Economy (SZ)(广东人工智能与数字经济实验室);The Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学)
Reasoning as an Attack Surface: Adaptive Evolutionary CoT Jailbreaks for LLMs
Nanyang Technological University, Singapore(南洋理工大学);The University of Hong Kong, Hong Kong, China(香港大学);The Hong Kong Polytechnic University, Hong Kong, China(香港理工大学);Zhejiang University, Zhejiang, China(浙江大学);Renmin University of China, Beijing, China(中国人民大学);Sun Yat-sen University, Guangdong, China(中山大学);Northeastern University(东北大学);Hebei Key Laboratory of Data Science and Knowledge Management(河北省数据科学与知识管理重点实验室)
CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
School of Computer Science & Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(计算机科学与工程学院,新一代人工智能技术及其交叉应用关键实验室(东南大学),教育部,中国);Data Science & AI Department at Faculty of IT, Monash University, Australia(信息学院数据科学与人工智能部门,墨尔本大学,澳大利亚)
MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究所);Nankai University(南开大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州));School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院)
CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizers
Hanoi University of Science and Technology(河内理工大学);Oraichain Labs Inc.(Oraichain实验室公司);ByteDance Inc(字节跳动公司);VNU University of Engineering and Technology(越南国家大学工程技术大学);Monash University(莫纳什大学);University of Oregon(俄勒冈大学)
Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Artificial Intelligence and Data Science, USTC(中国科学技术大学人工智能与数据科学学院);Kuaishou Technology(快手科技);College of Future Information Technology, Fudan University(复旦大学未来信息技术学院)
IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking
Institute of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Information Research Center of Military Science, PLA Academy of Military Science(军事科学院军事科学信息研究中心)
When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo(宁波数字孪生研究院,东方理工高等研究院,宁波);Viterbi School of Engineering, University of Southern California(南加州大学维特比工程学院);The Hong Kong Polytechnic University(香港理工大学);LMU Munich(慕尼黑大学);Saarland University(萨尔大学)