思维链（Chain-of-Thought, CoT）解码技术的最新发展，使得大语言模型（Large Language Models, LLMs）能够为复杂问题求解生成显式的逻辑推理路径。但研究表明，这些路径并非总是经过审慎推导且最优的。思维树（Tree-of-Thought, ToT）方法采用树搜索来广泛探索推理空间，找到CoT解码可能遗漏的更优推理路径，然而这种审慎推导的代价是推理复杂度大幅提升。本研究表明，利用ToT构建的搜索树微调LLMs，可让CoT达到相近甚至更优的性能，从而避免沉重的推理负担。该效果通过偏好优化链（Chain of Preference Optimization, CPO）实现：借助树搜索过程中的内在偏好信息微调LLMs，使CoT推理路径的每一步都与ToT的推理路径对齐。大量实验结果显示，CPO显著提升了LLM在问答、事实验证、算术推理等各类复杂问题求解上的表现，证明了其有效性。我们的代码可在https://github.com/sail-sg/CPO获取。
英文摘要：
The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at https://github.com/sail-sg/CPO.
Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；China Mobile Information Technology Center（中国移动信息技术中心）
OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
UC San Diego（加州大学圣迭戈分校）；The University of New South Wales（新南威尔士大学）；East China Normal University（华东师范大学）；Adobe Research（奥多比研究中心）；CSIRO’s Data61（联邦科学与工业研究组织数据61部门）
Less is More: Improving LLM Alignment via Preference Data Selection
University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；Massachusetts Institute of Technology（麻省理工学院）；Alibaba Cloud Computing（阿里云计算）；MoE Key Lab of BIPC, University of Science and Technology of China（中国科学技术大学MoE关键实验室）
Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）；University College London（伦敦大学学院）
Multi-Response Preference Optimization with Augmented Ranking Dataset
INMED DATA；Yonsei University（延世大学）；Asan Medical Center（峨山医院）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Institute for Life Sciences（峨山生命科学研究所）
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）
Large Language Models Can Better Understand Knowledge Graphs Than We Thought
Southeast University（东南大学）；The University of New South Wales（新南威尔士大学）；Monash University（莫纳什大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Southeast University（东南大学）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) Ministry of Education（新一代人工智能技术及其交叉应用国家重点实验室）；China Mobile Research Institute（中国移动研究院）；Monash University（墨尔本大学）；University of Manchester（曼彻斯特大学）；University of Edinburgh（爱丁堡大学）
APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（计算与智能学院，哈尔滨工业大学，深圳，中国）；Huawei Cloud Computing Technologies Co., Ltd.（华为云计算技术有限公司）；School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China（智能科学与工程学院，哈尔滨工业大学，深圳，中国）
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies(PKU), MOE, China（教育部高可信软件技术重点实验室）；Beijing Jiaotong University（北京交通大学）
