Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT3. On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs. We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA task, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can incorporate a symbolic information retrieval within our decomposition framework, leading to improved performance on both tasks. Datasets, Code and Prompts available at https://github.com/allenai/DecomP.
Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models
Center for Information and Language Processing (CIS)（信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；ScaDS.AI and TU Dresden（ScaDS.AI 和 梅克伦堡-前波美拉尼亚技术大学）；Department of Statistics, LMU Munich（统计学系，慕尼黑大学）；University of Maryland, College Park（马里兰大学 College Park 分校）
Center for Information and Language Processing（信息与语言处理中心）；LMU Munich（慕尼黑大学）
Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.
On Conditional and Compositional Language Model Differentiable Prompting
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Amazon Alexa（亚马逊Alexa）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Models
Laboratoire de traitement et transport de l'information, Université Sorbonne Paris Nord, France（信息处理与传输实验室，索邦巴黎北大学，法国）；Institute of Communication Studies, National Yang Ming Chiao Tung University, Taiwan（传播学研究所，国立阳明交通大学，台湾）
Heterogeneous LLM Methods for Ontology Learning (Few-Shot Prompting, Ensemble Typing, and Attention-Based Taxonomies)
Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究院）；BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory, University of Sharjah（沙迦大学生物医学引导人工智能实验室）；Ivannikov Institute for System Programming of the Russian Academy of Sciences（俄罗斯科学院伊万诺夫系统编程研究所）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫莫斯科大学）；AIRI Institute（AIRI研究所）
Electronic Circuit Principles of Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Central South University（中南大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；ByteDance Seed (China)（字节跳动种子（中国））
University of Rochester（罗切斯特大学）；Tencent AI Lab（腾讯人工智能实验室）；University of Pennsylvania（宾夕法尼亚大学）
Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the need to utilize external knowledge to assist LLMs. Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the dec…
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is dec…
Prompting large language models has enabled significant recent progress in multi-step reasoning over text. However, when applied to text generation from semi-structured data (e.g., graphs or tables), these methods typically suffer from low semantic coverage, hallucination, and logical inconsistency. We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. MURMUR is a best-first search method that generates reasoning paths using: (1) neural and symbolic modules with specific linguistic and logical skills, (2) a grammar whose pr…
Many NLP learning tasks can be decomposed into several distinct sub-tasks, each associated with a partial label. In this paper we focus on a popular class of learning problems, sequence prediction applied to several sentiment analysis tasks, and suggest a modular learning approach in which different sub-tasks are learned using separate functional modules, combined to perform the final task while sharing information. Our experiments show this approach helps constrain the learning process and can alleviate some of the supervision efforts.
Decomposed Opinion Summarization with Verified Aspect-Aware Modules
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；School of Informatics, The University of Edinburgh（爱丁堡大学信息学院）
