In this work, we study in-context teaching (ICT), where a teacher provides in-context example rationales to teach a student to reason over unseen cases. Human teachers are usually required to craft in-context demonstrations, which are costly and have high variance. We ask whether a large language model (LLM) can serve as a more effective in-context teacher for itself or other LLMs, compared to humans. Inspired by the Encoding Specificity Hypothesis from human episodic memory, we hypothesize that in-context exemplars crafted by the teacher should match the training data of the student. This hypothesis motivates us to propose Self-Explain where an LLM's self-elicited explanations are used as in-context demonstrations for prompting it as they are generalized from the model's training examples. Self-Explain is shown to significantly outperform using human-crafted exemplars and other baselines.
Furthermore, we reveal that for ICT, rationales from different teacher LLMs or human experts that more resemble the student LLM's self-explanations are better in-context demonstrations. This supports our encoding specificity hypothesis. We then propose Teach-Back that aligns a teacher LLM with the student to enhance the ICT performance. For example, Teach-Back enables a 7B model to teach the much larger GPT-3.5 in context, surpassing human teachers by around 5% in test accuracy on medical question answering.
Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
College of Computer Science and Technology（计算机科学与技术学院）；Alibaba Cloud Computing（阿里云计算）；School of Artificial Intelligence（人工智能学院）；Department of Mathematics（数学系）
RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Inc.（华为公司）
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
Long-Chain Reasoning Distillation via Adaptive Prefix Alignment
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学计算机科学与技术系）；Alibaba Group, China（阿里巴巴集团）
Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学人工智能研究院计算机科学与技术系）；School of Computer Science, Beijing University of Posts and Telecommunications, China（北京邮电大学计算机科学学院）
"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Nankai University（南开大学）；The Hong Kong University of Science and Technology(Guangzhou)（香港科技大学（广州））；School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（软件学院，人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
Tsinghua University（清华大学）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）；Huazhong University of Science and Technology（华中科技大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
Tongji University（同济大学）；Hong Kong Polytechnic University（香港理工大学）；Shanghai AI Lab（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）
Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）；Carnegie Mellon University（卡内基梅隆大学）；Rutgers University（罗格斯大学）；The Chinese University of Hong Kong（香港中文大学）
Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models
Einstein Global Advanced Technologies for Equity (E-GATE), Hospital Israelita Albert Einstein（以色列阿尔伯特·爱因斯坦医院 爱因斯坦全球公平先进技术中心（E-GATE））；Institute of Science and Technology Itaú (ICTi), Itaú Unibanco（伊塔乌联合银行 伊塔乌科学技术研究所（ICTi））；School of Sciences, São Paulo State University（圣保罗州立大学 理学院）
In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
Princeton Language and Intelligence（普林斯顿语言与智能研究所）；Kempner Institute, Harvard（哈佛凯普纳研究所）；Microsoft Research, New York（微软研究院（纽约））；University of Pennsylvania（宾夕法尼亚大学）
Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
