受人类认知启发，提出Markov Chain of Thought (MCoT)框架，将多步推理步骤压缩为简化问题，在保持准确率的同时显著提升了大型语言模型数学推理的计算效率。
摘要：
多步的Chain of Thought (CoT)受益于推理步骤的逻辑结构和任务特定动作，显著增强了大型语言模型的数学推理能力。随着长CoT的普及，推理步骤的数量超出了可管理的token限制，并导致了更高的计算需求。受人类认知基本逻辑“推导，然后缩减”的启发，我们将标准的多步CoT概念化为一种新颖的Markov Chain of Thought (MCoT)。在本研究中，我们考虑数学推理任务，将每个推理步骤定义为文本并附带Python代码片段。为促进更长的推理路径，通过与代码解释器的交互实现了自我纠错。我们的MCoT旨在将先前的推理步骤压缩为一个简化的问题，从而在不依赖冗长KV缓存的情况下实现高效的下一步推理。在实验中，我们整理了$\ exttt{MCoTInstruct}$数据集，实证结果表明，MCoT不仅显著提升了效率，还保持了相当的准确率。尽管仍有诸多待探索之处，这项工作为探索LLM的长CoT推理能力铺平了道路。代码可在https://github.com/james-yw/Markov-Chain-of-Thought获取。
英文摘要：
Chain of Thought (CoT) of multi-step benefits from the logical structure of the reasoning steps and task-specific actions, significantly enhancing the mathematical reasoning capabilities of large language models. As the prevalence of long CoT, the number of reasoning steps exceeds manageable token limits and leads to higher computational demands. Inspired by the fundamental logic of human cognition, "derive, then reduce", we conceptualize the standard multi-step CoT as a novel Markov Chain of Thought (MCoT). In this study, we consider the mathematical reasoning task, defining each reasoning step as text accompanied by a Python code snippet. To facilitate a longer reasoning path, self-correction is enabled through interactions with the code interpreter. Our MCoT aims to compress previous reasoning steps into a simplified question, enabling efficient next-step inference without relying on a lengthy KV cache. In our experiments, we curate the $\texttt{MCoTInstruct}$ dataset, and the empirical results indicate that MCoT not only significantly enhances efficiency but also maintains comparable accuracy. While much remains to be explored, this work paves the way for exploring the long CoT reasoning abilities of LLMs. The code is available at https://github.com/james-yw/Markov-Chain-of-Thought
Distilling Mathematical Reasoning Capabilities into Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）
LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought
Shanghai Business School（上海商学院）；Learnable.AI Inc.（Learnable.AI公司）；Centre for Frontier AI Research, A*STAR（前沿人工智能研究中心，A*STAR）；Institute of High-Performance Computing, A*STAR（高性能计算研究所，A*STAR）；The Hong Kong Polytechnic University（香港理工大学）；Microsoft Research Asia（微软亚洲研究院）
Cognitive Loop of Thought: Reversible Hierarchical Markov Chain for Efficient Mathematical Reasoning
School of Computer Science and Technology, East China Normal University（1 计算机科学与技术学院，华东师范大学）；School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（2 电子电气工程学院，上海工程技术大学）
Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；The Chinese University of Hong Kong（香港中文大学）；Monash University（墨尔本大学）；Independent Researcher（独立研究者）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Alibaba Group（阿里巴巴集团）；Computational Bioscience Research Center, KAUST（阿卜杜拉国王科技大学计算生物科学研究中心）
DAG-Math: Graph-of-Thought Guided Mathematical Reasoning in LLMs
Department of Statistics, University of Warwick（沃里克大学统计系）；Google DeepMind（谷歌DeepMind）；Department of Applied Mathematics, Hong Kong Polytechnic University（香港理工大学应用数学系）；School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University（上海交通大学数学科学学院）
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Center for Data Science, New York University（纽约大学数据科学中心）
Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated …
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Baidu Inc（百度公司）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University（吉林大学教育部符号计算与知识工程重点实验室）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室）
Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning
Tencent BAC（腾讯BAC）；Shenzhen International Graduate School, Tsinghua University（深圳国际研究生院，清华大学）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学学院）
