arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.30128cs.AIcs.CL
收藏

冗长的思维链真的有用吗?来自分布内证据表明,内容而非长度才是关键

Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

Fin AI Research(Fin AI研究)

Wenlong Wang, Fergal Reid

AI总结:

通过分布内采样和受控干预实验,发现思维链中额外标记的长度不影响推理准确性,真正起作用的是标记携带的推理和验证内容。

摘要:

思维链提示提升了大型语言模型的推理能力,但其来源存在争议:中间步骤之所以有帮助,是因为它们携带了有用的语义内容,还是因为模型在确定答案之前,通过条件化更多标记获得了额外的计算?我们提供了两方面的证据。首先,在分布内:我们在同一问题上对每个模型进行重复采样,并将遵循相同推理计划的较短自然生成与较长自然生成配对,因此没有重写,两个轨迹都是真正的分布内。在25个模型中,额外的标记对每个独立训练推理者的准确性几乎没有改变,而对多余标记的盲分析表明,存在的增益与验证和检查内容相关,而非冗长本身。其次,作为受控干预,我们询问两个表达相同语义内容(通过有向无环图等价验证的相同事实、操作和中间值)的轨迹,当其中一个更冗长时,是否会产生不同的结果,采用双验证器设计,涉及四个目标和八个基准,使用数字遮蔽完成和分层自助置信区间。冗长的轨迹确实提高了准确性(在至少一个验证器下,32个基准-目标单元中有25个为正),但效果适中(通常为1-4个百分点),并且取决于冗长文本的质量,而不仅仅是其长度。在最大数字遮蔽下,效果被放大(在四个算术基准上中位数增加3.24倍),而长度匹配的非推理填充则无法恢复任何效果。两条证据汇聚:重要的是额外标记所做的事情(它们携带的推理和验证内容),而不是它们的数量,这种图景既不能由纯前向计算解释,也不能由纯语义内容解释。

英文摘要:

Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer? We bring two lines of evidence to bear. First, in distribution: we repeatedly sample each model on the same question and pair a shorter with a longer of its own natural generations that follow the same reasoning plan, so nothing is rewritten and both traces are genuinely in-distribution. Across 25 models the extra tokens leave accuracy essentially unchanged for every independently-trained reasoner, and a blind analysis of the surplus tokens shows that what gain exists elsewhere tracks validation- and checking-content, not verbosity per se. Second, as a controlled intervention, we ask whether two traces expressing the same semantic content (the same facts, operations, and intermediate values, verified through directed acyclic graph equivalence) produce different outcomes when one is more verbose, using a dual-validator design across four targets and eight benchmarks with number-redacted completion and stratified bootstrap confidence intervals. Verbose traces do improve accuracy (25 of 32 benchmark-target cells are positive under at least one validator), but the effects are modest (typically 1-4 points) and depend on the quality of the verbose prose, not merely its length. Under maximum numerical redaction the effect is amplified (median 3.24x across four arithmetic benchmarks), and length-matched non-reasoning filler recovers none of it. Both lines converge: what matters is what the extra tokens do (the reasoning and validation content they carry), not how many there are, a picture neither a pure forward-pass-compute nor a pure semantic-content account fully explains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑