提示科学报告2：思维链提示的价值正在下降
Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
AI总结：
本报告通过严格测试评估CoT提示，发现其收益高度依赖任务与模型，对具备内置推理能力的新模型价值有限，却显著增加token与时间成本。
摘要：
这是一系列简短报告中的第二篇，旨在通过严格测试，帮助商业、教育和政策领导者理解使用AI的技术细节。在本报告中，我们研究了Chain-of-Thought（CoT，思维链）提示，这是一种鼓励大语言模型（LLM）“逐步思考”的技术（Wei等人，2022）。CoT是一种被广泛采用的、用于改进推理任务的方法；然而，我们的研究结果揭示了其有效性更为微妙的一面。我们证明了两点：第一，Chain-of-Thought提示的有效性会因任务和模型类型的不同而有很大差异。对于非推理模型，CoT通常会小幅提升平均性能，尤其是在模型默认情况下不会内在地进行逐步处理时。然而，CoT可能会给答案带来更多变化，有时会在模型原本能够答对的问题上引发偶发错误。我们还发现，许多近期模型即使没有被要求，也会执行某种形式的CoT推理；对于这些模型，执行CoT的请求几乎没有影响。与直接回答相比，执行CoT通常需要多得多的token（从而增加成本和时间）。第二，对于具有显式推理能力的模型，CoT提示在答案准确性方面往往只能带来边际收益（如果有的话），但它会显著增加生成响应所需的时间和token。
英文摘要：
This is the second in a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we investigate Chain-of-Thought (CoT) prompting, a technique that encourages a large language model (LLM) to "think step by step" (Wei et al., 2022). CoT is a widely adopted method for improving reasoning tasks, however, our findings reveal a more nuanced picture of its effectiveness. We demonstrate two things: - The effectiveness of Chain-of-Thought prompting can vary greatly depending on the type of task and model. For non-reasoning models, CoT generally improves average performance by a small amount, particularly if the model does not inherently engage in step-by-step processing by default. However, CoT can introduce more variability in answers, sometimes triggering occasional errors in questions the model would otherwise get right. We also found that many recent models perform some form of CoT reasoning even if not asked; for these models, a request to perform CoT had little impact. Performing CoT generally requires far more tokens (increasing cost and time) than direct answers. - For models designed with explicit reasoning capabilities, CoT prompting often results in only marginal, if any, gains in answer accuracy. However, it significantly increases the time and tokens needed to generate a response.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
