用于测试时扩展的结果奖励模型的逻辑推理
Logical Reasoning with Outcome Reward Models for Test-Time Scaling
AI总结：
本文提出用于演绎逻辑推理的结果奖励模型，并通过回声生成技术扩展训练数据中的错误类型，在FOLIO、JustLogic和ProverQA数据集上提升了多个LLMs的推理性能。
摘要：
逻辑推理是评估大型语言模型（LLMs）能力的一个关键基准，因为它反映了模型从给定前提推导出有效结论的能力。虽然测试时扩展与专用的结果奖励模型或过程奖励模型的结合，为提升LLMs在复杂推理任务中的性能开辟了新途径，但这一空间在演绎逻辑推理中尚未得到充分探索。我们提出了一组用于演绎推理的结果奖励模型（Outcome Reward Models，ORMs）。为了训练这些ORMs，我们主要使用单样本和多样本的思维链（Chain-of-Thought，CoT）来生成数据。此外，我们提出了一种新颖的策略，以进一步扩展ORM训练数据集中所覆盖的错误类型。具体来说，我们提出了一种回声生成技术，该技术利用LLMs倾向于反映提示中错误假设的特性，来提取额外的训练数据，从而覆盖此前未被探索的错误类型。虽然标准的CoT链可能包含推理者容易犯的错误，但回声策略会刻意引导模型走向错误推理。我们表明，在CoT和回声增强数据上训练的ORMs，在FOLIO、JustLogic和ProverQA数据集上，针对四个不同的LLMs，均展现出更好的性能。
英文摘要：
Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
