Although large language models (LLMs) have demonstrated adeptness in a range of tasks, they still lag behind human learning efficiency. This disparity is often linked to the inherent human capacity to learn from basic examples, gradually generalize and handle more complex problems, and refine their skills with continuous feedback. Inspired by this, this paper introduces YODA, a novel teacher-student progressive learning framework that emulates the teacher-student education process to improve the efficacy of model fine-tuning. The framework operates on an interactive \textit{basic-generalized-harder} loop. The teacher agent provides tailored feedback on the student's answers, and systematically organizes the education process. This process unfolds by teaching the student basic examples, reinforcing understanding through generalized questions, and then enhancing learning by posing questions with progressively enhanced complexity. With the teacher's guidance, the student learns to iteratively refine its answer with feedback, and forms a robust and comprehensive understanding of the posed questions. The systematic procedural data, which reflects the progressive learning process of humans, is then utilized for model training. Taking math reasoning as a testbed, experiments show that training LLaMA2 with data from YODA improves SFT with significant performance gain (+17.01\% on GSM8K and +9.98\% on MATH). In addition, we find that training with curriculum learning further improves learning robustness.
Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion
Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院多模态认知计算安徽省重点实验室）；Department of Automation, Tsinghua University（清华大学自动化系）
Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training
National University of Singapore（新加坡国立大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；The University of Sydney（悉尼大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SIGS, Tsinghua University（清华大学信息科学与技术学院）；Southern University of Science and Technology（南方科技大学）；Guangdong Laboratory of AI and Digital Economy (SZ)（广东人工智能与数字经济实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Peking University（北京大学）
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室（东南大学），教育部，中国）
SOD: Step-wise On-policy Distillation for Small Language Model Agents
Zhejiang University（浙江大学）；Large Language Model Department, Tencent（腾讯大语言模型部门）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
