REFINER:中间表示上的推理反馈
REFINER: Reasoning Feedback on Intermediate Representations
AI总结:
REFINER通过评论模型提供结构化反馈,迭代改进语言模型生成的中间推理步骤,在三个推理任务上显著优于基线,且无需微调推理器即可提升GPT-3.5/ChatGPT的推理性能。
摘要:
语言模型(LMs)近期通过显式生成中间推理步骤(如思维链提示)在推理任务上表现出显著性能。然而,这些中间推理步骤可能是不恰当的从初始上下文推导出的结论,并导致错误的最终预测。在此,我们介绍REFINER,一个用于微调语言模型以显式生成中间推理步骤的框架,同时与一个提供推理自动反馈的评论模型交互。具体而言,评论者提供结构化反馈,推理语言模型利用该反馈迭代改进其中间论据。对REFINER在三个不同推理任务上的实证评估显示,与同等规模的基础语言模型相比有显著改进。此外,当使用GPT-3.5或ChatGPT作为推理器时,训练好的评论者无需微调推理器即可显著提升推理能力。最后,我们的评论模型无需昂贵的人工参与循环数据即可训练,但在推理时可以用人类替代。
英文摘要:
Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences, e.g., chain-of-thought prompting. However, these intermediate inference steps may be inappropriate deductions from the initial context and lead to incorrect final predictions. Here we introduce REFINER, a framework for finetuning LMs to explicitly generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. Specifically, the critic provides structured feedback that the reasoning LM uses to iteratively improve its intermediate arguments. Empirical evaluations of REFINER on three diverse reasoning tasks show significant improvements over baseline LMs of comparable scale. Furthermore, when using GPT-3.5 or ChatGPT as the reasoner, the trained critic significantly improves reasoning without finetuning the reasoner. Finally, our critic model is trained without expensive human-in-the-loop data but can be substituted with humans at inference time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。