在 RLHF 中利用步骤信号增强演绎推理
Boosting Deductive Reasoning with Step Signals In RLHF
AI总结:
本文提出基于形式逻辑的自动化数据生成方法 MuseD,通过可控复杂度的多步演绎推理数据开展 RLHF 训练,显著提升模型在域内与域外任务中的逻辑推理能力。
摘要:
逻辑推理是大型语言模型(LLM)应对复杂问题的一项关键任务。在各类推理任务中,多步推理尤其具有挑战性。基于形式逻辑理论,我们开发了一种用于演绎推理数据的自动化方法 Multi-step Deduction(MuseD)。MuseD 使我们能够构建用于多步推理的训练和测试数据集。我们的生成方法可以控制所生成指令的复杂度,从而便于在不同难度级别上训练和评估模型。通过 RLHF 训练,我们的训练数据在域内和域外推理任务中均显著提升了模型的逻辑能力。此外,我们还进行了测试,以评估多种模型的多步推理能力。
英文摘要:
Logical reasoning is a crucial task for Large Language Models (LLMs), enabling them to tackle complex problems. Among reasoning tasks, multi-step reasoning poses a particular challenge. Grounded in the theory of formal logic, we have developed an automated method, Multi-step Deduction (MuseD), for deductive reasoning data. MuseD has allowed us to create training and testing datasets for multi-step reasoning. Our generation method enables control over the complexity of the generated instructions, facilitating training and evaluation of models across different difficulty levels. Through RLHF training, our training data has demonstrated significant improvements in logical capabilities for both in-domain of out-of-domain reasoning tasks. Additionally, we have conducted tests to assess the multi-step reasoning abilities of various models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。