Strategies such as chain-of-thought prompting improve the performance of large language models (LLMs) on complex reasoning tasks by decomposing input examples into intermediate steps. However, it remains unclear how to apply such methods to reason over long input documents, in which both the decomposition and the output of each intermediate step are non-trivial to obtain. In this work, we propose PEARL, a prompting framework to improve reasoning over long documents, which consists of three stages: action mining, plan formulation, and plan execution. More specifically, given a question about a long document, PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE, FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain the answer. Each stage of PEARL is implemented via zero-shot or few-shot prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate PEARL on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts. PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance. Overall, PEARL is a first step towards leveraging LLMs to reason over long documents.
GPT is becoming a Turing machine: Here are some ways to program it
Fred Hutchinson Cancer Research Center(弗雷德·哈钦森癌症研究中心);University of California San Diego(加利福尼亚大学圣迭戈分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Microsoft Research(微软研究院)
Large Language Models as Code Executors: An Exploratory Study
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Xinjiang University(新疆大学);University of Melbourne(墨尔本大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);IBM Research(IBM研究院);Nantong University(南通大学);Alibaba(阿里巴巴)
Automated Visualization Code Synthesis via Multi-Path Reasoning and Feedback-Driven Optimization
AI Research, Enhans, Seoul, South Korea Innovation \& Technology, KAIST, Daejeon, South Korea Department of Computer Science, University of California, Berkeley, CA, United States Department of Electrical
CLadder: Assessing Causal Reasoning in Language Models
MPI for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所,蒂宾根);ETH Zürich(苏黎世联邦理工学院);IIT Kharagpur(印度理工学院卡拉格普尔分校);University of Hong Kong(香港大学);University of Washington(华盛顿大学)
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
IDEA Research(IDEA研究院);International Digital Economy Academy(国际数字经济学院);Xiamen University(厦门大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Microsoft Research Asia(微软亚洲研究院);University of Southern California(南加州大学);The Hong Kong University of Science and Technology(香港科技大学)
A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis
Department of Electrical and Computer Engineering, University of Miami(迈阿密大学电气与计算机工程系);Department of Civil and Architectural Engineering, University of Miami(迈阿密大学土木与建筑工程系);College of Civil Engineering, Hunan University(湖南大学土木学院);Department of Computer Science, University of Illinois Chicago(伊利诺伊大学芝加哥分校计算机科学系);Center for Catastrophe Modeling and Resilience, Lehigh University(莱斯大学灾难建模与韧性中心);School of Architecture, University of Miami(迈阿密大学建筑学院)
CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China(华中科技大学计算机科学与技术学院);Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China(华中科技大学光电实验室);Department of Computer Science, Bowling Green State University, Ohio, USA(Bowling Green State University 计算机科学系);Department of Computer Science, Aalborg University, Aalborg, Denmark(Aalborg University 计算机科学系);Huawei Technologies Co., Ltd, Shenzhen, China(华为技术有限公司)
LitLLMs, LLMs for Literature Review: Are we there yet?
ServiceNow Research(ServiceNow研究院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);HEC Montreal(蒙特利尔高等商学院);University of Waterloo(滑铁卢大学);University of British Columbia(不列颠哥伦比亚大学);Polytechnique Montreal(蒙特利尔理工学院)
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
University of Houston(休斯顿大学);Alexandria University(亚历山大大学);Bogazici University(博阿齐奇大学);Ankara Yıldırım Beyazıt University(安卡拉耶尔德勒姆贝亚泽特大学);ESI, Algiers(阿尔及尔ESI);University of Algiers 1(阿尔及尔第一大学);York St John University London(约克圣约翰大学伦敦校区);Air University, Islamabad(伊斯兰堡空军大学);Mohammed Premier University(穆罕默德第一大学);Universidad Católica San Pablo(圣巴勃罗天主教大学);Busitema University(布西特马大学);University of Laghouat(拉格瓦特大学);University of Tunis El Manar(突尼斯埃尔马纳尔大学);Tribhuvan University(特里布万大学);IIT Roorkee(印度理工学院鲁尔基分校);Shobhit Institute of Engineering and Technology(肖比特工程技术学院);MIT World Peace University(MIT世界和平大学);University of Kinshasa(金沙萨大学);University of Bertoua(贝尔图阿大学);University of the Witwatersrand(威特沃特斯兰德大学)
ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Institute of Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究所)