让推理发挥作用：衡量并提升思维链推理的忠实度
Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
AI总结：
针对大语言模型思维链推理中最终答案与推理步骤忠实度不足的问题，提出包含双模块的FRODO框架，通过因果奖励与反事实偏好目标提升推理忠实度，在多个基线上取得更优效果与更强泛化性。
摘要：
大语言模型（LLM）在被要求先逐步推理再回答问题时表现更优，但目前尚不清楚模型的最终答案在多大程度上忠实于其给出的推理步骤。本文对12个LLM开展因果中介分析，探究LLM生成的中间推理步骤对最终输出的影响，发现LLM在生成答案时并不会可靠地利用其中间推理步骤。\n为解决这一问题，我们提出FRODO框架，用于定制小型语言模型，使其生成正确的推理步骤并基于这些步骤进行鲁棒推理。FRODO包含两个模块：推理生成模块通过隐式因果奖励函数学习生成正确的推理步骤；推理执行模块通过反事实与因果偏好目标，学习基于这些中间推断结果进行忠实推理。\n实验表明，FRODO显著优于4个强基线模型。此外，FRODO提升了推理语言模型的鲁棒性和泛化能力，在分布外测试集上取得了更高的性能。最后我们还发现，与标准监督微调相比，FRODO生成的推理依据与其最终答案预测的忠实度更高。
英文摘要：
Large language models (LLMs) have been shown to perform better when asked to reason step-by-step before answering a question. However, it is unclear to what degree the model's final answer is faithful to the stated reasoning steps. In this paper, we perform a causal mediation analysis on twelve LLMs to examine how intermediate reasoning steps generated by the LLM influence the final outcome and find that LLMs do not reliably use their intermediate reasoning steps when generating an answer. To address this issue, we introduce FRODO, a framework to tailor small-sized LMs to generate correct reasoning steps and robustly reason over these steps. FRODO consists of an inference module that learns to generate correct reasoning steps using an implicit causal reward function and a reasoning module that learns to faithfully reason over these intermediate inferences using a counterfactual and causal preference objective. Our experiments show that FRODO significantly outperforms four competitive baselines. Furthermore, FRODO improves the robustness and generalization ability of the reasoning LM, yielding higher performance on out-of-distribution test sets. Finally, we find that FRODO's rationales are more faithful to its final answer predictions than standard supervised fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
