通过多听众软执行平衡忠实与性能
Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution
AI总结：
REMUL通过多听众强化学习方法提升推理忠实性与性能，改进多个基准的忠实性指标和准确性。
摘要：
链式思维（CoT）推理有时无法忠实反映大型语言模型（LLM）的真实计算，这阻碍了其在解释LLM如何得出答案中的实用性。此外，优化推理中的忠实性和可解释性往往会降低任务性能。为了解决这一权衡问题并提高CoT的忠实性，我们提出了多听众推理执行（REMUL），一种多方强化学习方法。REMUL基于假设：其他方能够跟随的推理轨迹会更忠实。生成器模型生成推理轨迹，将其截断并传递给一组监听器模型，这些模型“执行”轨迹，继续轨迹以得到答案。生成器因生成清晰给监听器的推理而受奖励，通过掩码监督微调进行额外的正确性正则化，以应对忠实性与性能之间的权衡。在多个推理基准（BIG-Bench Extra Hard、MuSR、ZebraLogicBench和FOLIO）上，REMUL在忠实性三个指标——提示归因、曲线上的早期回答区域（AOC）和错误注入AOC——上持续且显著提高，同时提高了准确性。我们的分析发现，这些增益在训练领域上是稳健的，转化为可读性增益，并与更短和更直接的CoTs相关联。
英文摘要：
Chain-of-thought (CoT) reasoning sometimes fails to faithfully reflect the true computation of a large language model (LLM), hampering its utility in explaining how LLMs arrive at their answers. Moreover, optimizing for faithfulness and interpretability in reasoning often degrades task performance. To address this tradeoff and improve CoT faithfulness, we propose Reasoning Execution by Multiple Listeners (REMUL), a multi-party reinforcement learning approach. REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful. A speaker model generates a reasoning trace, which is truncated and passed to a pool of listener models who "execute" the trace, continuing the trace to an answer. Speakers are rewarded for producing reasoning that is clear to listeners, with additional correctness regularization via masked supervised finetuning to counter the tradeoff between faithfulness and performance. On multiple reasoning benchmarks (BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO), REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy. Our analysis finds that these gains are robust across training domains, translate to legibility gains, and are associated with shorter and more direct CoTs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
