arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14564cs.CL
收藏

PEARL:提示大型语言模型在长文档上规划并执行动作

PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Microsoft Research(微软研究院)

Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, Mohit Iyyer

AI总结:

本文提出PEARL提示框架,通过动作挖掘、计划制定和计划执行三阶段,利用GPT-4在长文档上规划并执行动作,显著提升了LLMs在长文档复杂推理任务上的表现。

摘要:

诸如思维链提示等策略通过将输入样本分解为中间步骤,提升了大型语言模型(LLMs)在复杂推理任务上的表现。然而,如何将此类方法应用于对长输入文档的推理仍不明确,因为在长文档中,分解过程以及每个中间步骤的输出都不易获取。在这项工作中,我们提出了PEARL,一个旨在改善长文档推理的提示框架,它包含三个阶段:动作挖掘、计划制定和计划执行。更具体地说,给定关于长文档的问题,PEARL将其分解为一系列动作(例如,SUMMARIZE、FIND_EVENT、FIND_RELATION),然后在文档上执行这些动作以获取答案。PEARL的每个阶段均通过LLMs(在我们的工作中为GPT-4)的零样本或少样本提示实现,仅需最少的人工输入。我们在QuALITY数据集的一个具有挑战性的子集上评估了PEARL,该子集包含需要在长叙事文本上进行复杂推理的问题。PEARL在此数据集上优于零样本和思维链提示,且消融实验表明PEARL的每个阶段对其性能都至关重要。总体而言,PEARL是利用LLMs对长文档进行推理的第一步。

英文摘要:

Strategies such as chain-of-thought prompting improve the performance of large language models (LLMs) on complex reasoning tasks by decomposing input examples into intermediate steps. However, it remains unclear how to apply such methods to reason over long input documents, in which both the decomposition and the output of each intermediate step are non-trivial to obtain. In this work, we propose PEARL, a prompting framework to improve reasoning over long documents, which consists of three stages: action mining, plan formulation, and plan execution. More specifically, given a question about a long document, PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE, FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain the answer. Each stage of PEARL is implemented via zero-shot or few-shot prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate PEARL on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts. PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance. Overall, PEARL is a first step towards leveraging LLMs to reason over long documents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑