自一致性正在失去优势:现代大语言模型中的边际效益递减与成本上升
Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
AI总结:
研究指出,随着模型增强,自一致性技术变得低效且可能降低性能,通过实验显示增加推理路径数量带来的准确率提升有限,而成本却显著增加,建议仅在单次可靠度不足的问题上使用多路径采样。
摘要:
自一致性——通过采样多个推理路径并选择最频繁的答案——最初设计用于语言模型频繁且不可预测出错的时代。本研究认为,随着模型能力增强,该技术变得越来越浪费资源,并可能在现代模型已能可靠解决的问题上降低性能。使用Gemini 2.5模型在HotpotQA和MATH-500上进行实验,结果显示增加采样路径数量带来的准确率提升极低——在HotpotQA上20次采样仅提升0.4%,在MATH-500上提升1.6%——而token成本几乎与采样次数成线性增长。关键发现是性能在早期趋于平缓,某些配置下在高采样次数时反而下降,表明当模型已能可靠解决问题时,额外路径引入噪声而非信号。随着模型规模扩大,推理成本上升,盲目使用自一致性难以成立。我们建议仅在明显超出模型单次可靠度的问题上保留多路径采样。
英文摘要:
Self-consistency -- sampling multiple reasoning paths and selecting the most frequent answer -- was designed for an era when language models made frequent, unpredictable errors. This study argues that the technique has become increasingly wasteful as models grow stronger, and may degrade performance on problems that modern models already solve reliably. Using Gemini 2.5 models on HotpotQA and MATH-500, we show that accuracy gains from increasing the number of sampled reasoning paths are minimal -- 0.4% on HotpotQA across 20 samples, and 1.6% on MATH-500 -- while token costs scale nearly linearly with sample count. Critically, performance plateaued early and in some configurations declined at high sample counts, suggesting that additional paths introduce noise rather than signal when models already solve problems reliably. As inference costs rise with model scale, indiscriminate self-consistency is difficult to justify. We recommend reserving multi-path sampling for problems that demonstrably exceed a model's single-pass reliability.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。