Distilling Reasoning Ability from Large Language Models with Adaptive Thinking
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
Chain of thought finetuning (cot-finetuning) aims to endow small language models (SLM) with reasoning ability to improve their performance towards specific tasks by allowing them to imitate the reasoning procedure of large language models (LLM) beyond simply predicting the answers. Most existing cot-finetuning methods adopt a pre-thinking mechanism, allowing the SLM to generate a rationale before providing an answer. This mechanism enables SLM to analyze and think about complex questions, but it also makes answer correctness highly sensitive to minor errors in rationale. Therefore, we propose a robust post-thinking mechanism to generate answers before rationale. Thanks to this answer-first setting, 1) the answer can escape from the adverse effects caused by minor errors in the rationale; 2) the rationale serves as an error amplifier to the answer, which makes the SLM focus on learning hard samples; 3) the inferring efficiency can also benefit from the setting since users can stop the generation right after answers are outputted when inference is conducted. However, although the post-thinking mechanism brings many advantages and improves the overall performance of SLM on specific tasks, it may lose the ability to think about the questions and decompose complex questions into simple sub-questions compared to pre-thinking mechanism. Therefore, a plug-and-play adaptive-thinking mechanism is proposed with the aid of the soft prompt tuning to integrate the merits of the pre-thinking mechanism and post-thinking mechanism, in which a perception module is introduced to adaptively prompt SLM answer or think first based on perceiving the complexity of the questions. Extensive experiments are conducted across 12 reasoning tasks and 2 representative language models to demonstrate the effectiveness of the proposed mechanism.
Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models
Zhejiang University(浙江大学);School of Public Health, Zhejiang University(浙江大学公共卫生学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Liangzhu Laboratory and Institute of Wenzhou, Zhejiang University(浙江大学良渚实验室与温州研究院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Institute of Digital Twin, EIT(数字孪生研究院);Saarland University(萨尔兰大学);Meituan Inc.(美团公司)
RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Huawei Inc.(华为公司)
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University(昌原大学);Korea University(韩国大学);Seoul National University(首尔国立大学);Korea Institute for Curriculum and Evaluation(韩国课程与评价研究院);Upstage;Indiana University Bloomington(印第安纳大学布卢明顿分校);Nanyang Technological University(南洋理工大学)
Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
Department of Computer Science and Engineering, The Ohio State University, USA(计算机科学与工程系,俄亥俄州立大学);Department of Biomedical Informatics, The Ohio State University, USA(生物医学信息学系,俄亥俄州立大学);Translational Data Analytics Institute, The Ohio State University, USA(转化数据分析研究所,俄亥俄州立大学)
ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection
National Key Laboratory for Novel Software Technology, Nanjing University, Jiangsu, China(新型软件技术国家重点实验室,南京大学,江苏,中国);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室,上海,中国);Department of Linguistics and Translation, City University of Hong Kong, Hong Kong, China(语言学与翻译系,香港城市大学,香港,中国);School of Mathematical Sciences, Dalian University of Technology, Liaoning, China(数学科学学院,大连理工大学,辽宁,中国)
Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
Distilling Mathematical Reasoning Capabilities into Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University College London, University of London(伦敦大学大学学院);Dalian University of Technology(大连理工大学)
Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models(北京大型模型研究关键实验室);Engineering Research Center of Next-Generation Intelligent Search(下一代智能搜索工程研究中心);College of Artificial Intelligence, Tianjin University of Science(天津科技大学人工智能学院)
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
University of California, Santa Barbara(加州大学圣芭芭拉分校);University of California, Santa Cruz(加州大学圣克ruz分校);University of California, Los Angeles(加州大学洛杉矶分校);Purdue University(普渡大学);LMSYS Org(LMSYS组织);Microsoft(微软)
Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);Ant Group(蚂蚁集团)
CAT: Confidence-Adaptive Thinking for Efficient Reasoning of Large Reasoning Models
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China(电子科技大学智能协同计算实验室);Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province(四川省泛在智能与可信服务重点实验室)