arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07078cs.AIcs.CL
收藏

CoTJudger: 一种基于图的框架,用于自动评估链式推理效率和冗余性在LRMs中

CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs

University of Science and Technology of China(科学技术大学);Shenzhen University of Advanced Technology(深圳先进技术大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究所,中国科学院);Southeast University(东南大学);Nanjing University(南京大学);Beihang University(北航);University of Manchester(曼彻斯特大学)

Siyi Li, Jiajun Shi, Shiwen Ni, Ge Zhang, Shuaimin Li, Shijian Wang, Zhoufutu Wen, Yizhi Li, Hamid Alinejad-Rokny, Jiaheng Liu, Min Yang, Wenhao Huang

AI总结:

CoTJudger通过构建依赖图提取最短有效路径,评估链式推理的效率与冗余,揭示模型中的冗余问题及失败模式。

摘要:

大型推理模型(LRMs)通过在回答前生成扩展的链式推理(CoT)轨迹展示了强大的性能。然而,这种范式往往导致过度推理:冗余计算和循环自我验证,这些会增加计算成本而不改善结果。现有的评估主要强调最终准确率或粗略的token计数,缺乏自动工具来区分必要的逻辑与结构冗余。我们引入了CoTJudger,一种基于图的框架,通过将自由形式的CoTs转换为有向依赖图,并提取所需的最短有效路径(SEP)来量化推理效率。这产生了一个可解释的效率信号——CoT中必要部分与结构冗余部分的比例——该信号在不同模型和任务之间具有可比性。评估21个LRMs,CoTJudger揭示了普遍的冗余性和反复出现的失败模式,包括验证痴迷和补偿性冗余。这些结果提供了一个实用的度量标准,用于区分推理能力与计算浪费,从而实现更精确的LRM效率评估和诊断。

英文摘要:

Large Reasoning Models (LRMs) have demonstrated strong performance by producing extended Chain-of-Thought (CoT) traces before answering. However, this paradigm often induces over-reasoning: redundant calculations and circular self-verification that increase computational cost without improving outcomes. Existing evaluations largely emphasize final accuracy or coarse token counts, and lack automated tools to separate essential logic from structural redundancy. We introduce CoTJudger, a graph-driven framework that quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path (SEP) needed to reach a correct solution. This yields an interpretable efficiency signal -- how much of a CoT is necessary versus structurally redundant -- that is comparable across models and tasks. Evaluating 21 LRMs, CoTJudger reveals pervasive redundancy and surfaces recurring failure modes, including verification obsession and compensatory redundancy. These results provide a practical metric for disentangling reasoning ability from computational waste, enabling more targeted evaluation and diagnosis of LRM efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑