思维链（Chain-of-thought, CoT）提示结合大语言模型已在众多自然语言处理任务中被证明有效，但设计能良好泛化到多样问题类型的提示颇具挑战，尤其是在数学应用题（math word problem, MWP）求解场景中。此外，通常存在大量覆盖多样性更好的训练数据，但缺少CoT标注，这限制了监督学习技术的应用。为解决这些问题，我们研究了两种在少样本提示场景下利用训练数据的方法：动态程序提示和程序蒸馏。我们的方法很大程度上受Gao等人（2022）的启发，他们提出用程序作为中间推理步骤来替代CoT。这种提示策略让我们能够在MWP求解中通过程序执行准确验证答案的正确性。我们的动态程序提示包括通过从大语言模型中采样正确程序来标注训练数据，而程序蒸馏则是让一个更小的模型适配带有程序标注的训练数据。我们在三个标准MWP数据集上的实验证明了这些方法的有效性，相较于之前的提示和微调基线取得了显著提升。我们的结果表明，利用大量训练数据可以提高提示的泛化能力，并提升微调后的小模型在MWP求解中的性能。
英文摘要：
Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging, especially in the context of math word problem (MWP) solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our approach is largely inspired by Gao et al., (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving. Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the program-annotated training data. Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning. Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of fine-tuned small models in MWP solving.
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University（新加坡管理大学）；Southwest Jiaotong University（西南交通大学）；Singapore University of Technology and Design（新加坡科技设计大学）；East China Normal University（华东师范大学）
Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Key Laboratory of Machine Perceptron (MOE), Peking University（北京大学机器感知教育部重点实验室）
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Alibaba Cloud Computing（阿里巴巴云计算）；Shanghai Jiao Tong University（上海交通大学）；College of Software, Zhejiang University（浙江大学软件学院）
Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（一般人工智能国家重点实验室，智能科学与技术学院，北京大学）
Metacognition-Enhanced Few-Shot Prompting With Positive Reinforcement
Institute of AI for Education, East China Normal University（华东师范大学教育人工智能研究院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Changning Mental Health Center（上海市长宁区精神卫生中心）
Determining the role of event arguments is a crucial subtask of event extraction. Most previous supervised models leverage costly annotations, which is not practical for open-domain applications. In this work, we propose to use global constraints with prompting to effectively tackles event argument classification without any annotation and task-specific training. Specifically, given an event and its associated passage, the model first creates several new passages by prefix prompts and cloze prompts, where prefix prompts indicate event type and trigger span, and cloze prompts connect each candi…
New York University（纽约大学）；University of California, San Diego（加州大学圣地亚哥分校）
Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, …
Diverge to Induce Prompting: Multi-Rationale Induction for Zero-Shot Reasoning
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan（国家台湾大学计算机科学与信息工程系）；Institute of Information Science, Academia Sinica, Taiwan（学术院信息科学研究所）；AI Research Center (AINTU), National Taiwan University, Taiwan（国家台湾大学人工智能研究中心（AINTU））
SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machine
International Digital Economy Academy（国际数字经济学院）；School of Software & Microelectronics, Peking University（北京大学软件与微电子学院）；National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机学院）；Data Science and Artificial Intelligence, Monash University（莫纳什大学数据科学与人工智能学院）；ModelTC
Self-Prompting Large Language Models for Zero-Shot Open-Domain QA
Shanghai Jiao Tong University（上海交通大学）；SJTU-Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎卓越工程师学院）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）
Large Language Models are affected by the phenomena of memorizing and forgetting their training data. But how do these vary by model size? We work towards this question by investigating how the model size affects the model's ability to discriminate a word's meaning in a given context. We introduce a dataset called DeltaWords, which evaluates a model's ability to follow instructions to select a sentence which replaces the target word with its antonym. We show a weak inverse scaling trend, where task accuracy degrades as model size increase, under extremely few-shot prompting regimes. We show th…
Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting
Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；Hong Kong Baptist University（香港浸会大学）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；College of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Prompting pre-trained language models has achieved impressive performance on various NLP tasks, especially in low data regimes. Despite the success of prompting in monolingual settings, applying prompt-based methods in multilingual scenarios has been limited to a narrow set of tasks, due to the high cost of handcrafting multilingual prompts. In this paper, we present the first work on prompt-based multilingual relation classification (RC), by introducing an efficient and effective method that constructs prompts from relation triples and involves only minimal translation for the class labels. W…
