arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.05074cs.CLcs.AI
收藏

DialCoT 遇见 PPO:在较小语言模型中分解并探索推理路径

DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models

East China Normal University(华东师范大学);Peking University(北京大学)

Chengcheng Han, Xiaowei Du, Che Zhang, Yixin Lian, Xiang Li, Ming Gao, Baoyuan Wang

AI总结:

针对思维链提示在参数少于100亿的较小语言模型上效果不佳的问题,提出DialCoT方法,通过对话格式生成中间推理步骤并结合PPO算法优化推理路径选择,在四个算术推理数据集上显著提升了性能。

摘要:

思维链(CoT)提示已被证明能有效增强具有至少 1000 亿参数的大型语言模型(LLMs)的推理能力。然而,将其应用于参数少于 100 亿的较小语言模型(SLMs)的推理任务时,不仅无效甚至有害。为了解决这一局限性,我们引入了对话引导思维链(DialCoT),它采用对话格式生成中间推理步骤,引导模型得出最终答案。此外,我们使用近端策略优化(PPO)算法优化模型的推理路径选择,进一步增强其推理能力。与以往方法相比,我们的方法具有几个优势。首先,我们将解决复杂推理问题的过程分解为一系列更简单的子问题,显著降低了任务难度,使其更适合 SLMs。其次,我们通过 PPO 算法优化模型的推理路径选择。我们在四个算术推理数据集上进行了全面实验,证明与最先进的竞争对手相比,我们的方法实现了显著的性能提升。

英文摘要:

Chain-of-Thought (CoT) prompting has proven to be effective in enhancing the reasoning capabilities of Large Language Models (LLMs) with at least 100 billion parameters. However, it is ineffective or even detrimental when applied to reasoning tasks in Smaller Language Models (SLMs) with less than 10 billion parameters. To address this limitation, we introduce Dialogue-guided Chain-of-Thought (DialCoT) which employs a dialogue format to generate intermediate reasoning steps, guiding the model toward the final answer. Additionally, we optimize the model's reasoning path selection using the Proximal Policy Optimization (PPO) algorithm, further enhancing its reasoning capabilities. Our method offers several advantages compared to previous approaches. Firstly, we transform the process of solving complex reasoning questions by breaking them down into a series of simpler sub-questions, significantly reducing the task difficulty and making it more suitable for SLMs. Secondly, we optimize the model's reasoning path selection through the PPO algorithm. We conduct comprehensive experiments on four arithmetic reasoning datasets, demonstrating that our method achieves significant performance improvements compared to state-of-the-art competitors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑