Distilling the thinking traces of a Large Language Model (LLM) with reasoning capabilities into a smaller model has been proven effective. Yet, there is a scarcity of work done on how model performances scale with the quantity of distillation data. In this work, we study the scaling trend of distilling competitive coding skills on two small non-reasoning LLMs. We validate the hypothesis that there is a $\textit{valley of code reasoning}$: downstream performance on competitive coding first drops as data quantity increases, then it steadily increases in a sharper-than-log-linear fashion. Having identified the trend, we further fine-tune the models at two different distillation stages on the same data to ground conclusions on their respective learning phases. We learn that across stages in the low and medium-low data regimes, small models benefit significantly from easier coding questions than from harder ones. We also find that, surprisingly, the correctness of outputs in training data makes no difference to distillation outcomes. Our work represents a step forward in understanding the training dynamics of code reasoning distillation outside intuition
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
College of Computer Science and Technology(计算机科学与技术学院);Alibaba Cloud Computing(阿里云计算);School of Artificial Intelligence(人工智能学院);Department of Mathematics(数学系)
Knowledge Distillation for Temporal Knowledge Graph Reasoning with Large Language Models
School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);School of Computing and Artificial Intelligence, Southwest Jiaotong University(西南交通大学计算机与人工智能学院);School of Information Science and Engineering, Chongqing Jiaotong University(重庆交通大学信息科学与工程学院);School of Information Engineering, Chang’an University(长安大学信息工程学院)
Long-Chain Reasoning Distillation via Adaptive Prefix Alignment
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系);Alibaba Group, China(阿里巴巴集团)
Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Meituan(美团);University of Science and Technology of China(中国科学技术大学)
MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究所);Nankai University(南开大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州));School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院)
Hán Dān Xué Bù (Mimicry) or Qīng Chū Yú Lán (Mastery)? A Cognitive Perspective on Reasoning Distillation in Large Language Models
School of Philosophy, Anhui University(安徽大学哲学学院);Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);Faculty of Education, University of Cambridge(剑桥大学教育学院);School of Foreign Studies, South China Normal University(华南师范大学外语学院);Division of Psychology and Language Sciences, University College London(伦敦大学学院心理学与语言科学系)
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University(吉林大学符号计算与知识工程重点实验室);Tongyi Lab, Alibaba Group(阿里集团通义实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning
Faculty of Computer and Information Science, University of Ljubljana, Slovenia(卢布尔雅那大学计算机与信息科学学院);ICAM - Advanced Environmental Research Institute, West University of Timisoara, Romania(蒂米șoara西大学先进环境研究所);Department of Computer Science, West University of Timisoara, Romania(蒂米șoa拉西大学计算机科学系)
Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)