arXiv 2310.135522023-10-23cs.CLcs.AI

大型语言模型上的自提示思维链用于开放域多跳推理

Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning

SJTU-Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学上海交大-巴黎精英工程师学院）；Shanghai Jiao Tong University（上海交通大学）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）

Jinyuan Wang, Junlong Li, Hai Zhao

备注：: Accepted by Findings of EMNLP2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出自动化框架SP-CoT，通过构建高质量ODMR数据集、自适应采样器和自提示推理提升LLMs的多跳推理能力，在大规模模型上超越SOTA，并将小规模模型零样本性能近乎翻倍。

摘要：

在开放域问答（ODQA）中，现有的大多数问题仅需基于常识进行单跳推理。为进一步扩展该任务，我们正式提出开放域多跳推理（ODMR），即在开放域环境下通过显式推理步骤来回答多跳问题。近期，大型语言模型（LLMs）在无需外部语料库辅助的ODQA中展现出重要价值。此外，思维链（CoT）提示通过手动或自动化范式极大提升了LLMs的推理能力。然而，现有的自动化方法缺乏质量保证，而手动方法受限于可扩展性不足和多样性差，阻碍了LLMs的能力发挥。本文提出自提示思维链（SP-CoT），这是一个由LLMs驱动、为LLMs服务并大规模生产高质量CoT的自动化框架。SP-CoT引入了高质量ODMR数据集的自动化生成流水线、用于上下文CoT选择的自适应采样器，以及基于上下文学习的自提示推理。在四个多跳问答基准上的广泛实验表明，提出的SP-CoT不仅在大规模（175B）LLMs上显著超越先前的SOTA方法，还将小规模（13B）LLMs的零样本性能几乎翻倍。进一步分析显示，SP-CoT在MuSiQue-Ans数据集上通过召回约50%的中间答案，展现出引发直接且简洁的中间推理步骤的卓越能力。

英文摘要：

In open-domain question-answering (ODQA), most existing questions require single-hop reasoning on commonsense. To further extend this task, we officially introduce open-domain multi-hop reasoning (ODMR) by answering multi-hop questions with explicit reasoning steps in open-domain setting. Recently, large language models (LLMs) have found significant utility in facilitating ODQA without external corpus. Furthermore, chain-of-thought (CoT) prompting boosts the reasoning capability of LLMs to a greater extent with manual or automated paradigms. However, existing automated methods lack of quality assurance, while manual approaches suffer from limited scalability and poor diversity, hindering the capabilities of LLMs. In this paper, we propose Self-prompted Chain-of-Thought (SP-CoT), an automated framework to mass-produce high quality CoTs of LLMs, by LLMs and for LLMs. SP-CoT introduces an automated generation pipeline of high quality ODMR datasets, an adaptive sampler for in-context CoT selection and self-prompted inference via in-context learning. Extensive experiments on four multi-hop question-answering benchmarks show that our proposed SP-CoT not only significantly surpasses the previous SOTA methods on large-scale (175B) LLMs, but also nearly doubles the zero-shot performance of small-scale (13B) LLMs. Further analysis reveals the remarkable capability of SP-CoT to elicit direct and concise intermediate reasoning steps by recalling $\sim$50\% of intermediate answers on MuSiQue-Ans dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。