This report examines the effectiveness of Chain-of-Thought (CoT) prompting in improving the multi-step reasoning abilities of large language models (LLMs). Inspired by previous studies \cite{Min2022RethinkingWork}, we analyze the impact of three types of CoT prompt perturbations, namely CoT order, CoT values, and CoT operators on the performance of GPT-3 on various tasks. Our findings show that incorrect CoT prompting leads to poor performance on accuracy metrics. Correct values in the CoT is crucial for predicting correct answers. Moreover, incorrect demonstrations, where the CoT operators or the CoT order are wrong, do not affect the performance as drastically when compared to the value based perturbations. This research deepens our understanding of CoT prompting and opens some new questions regarding the capability of LLMs to learn reasoning in context.
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. Th…
Shanghai Jiao Tong University（上海交通大学）；Amazon Web Services（亚马逊云科技）
Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstration…
Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning
School of Computer Science, Wuhan University（武汉大学计算机学院）；National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；School of Electronic Information Engineering, Henan Polytechnic Institute（河南工业职业技术学院电子信息工程学院）；JD Explore Academy（京东探索研究院）
CoEvoT: Co-Evolving Chain-of-Thought Prompting for Graph-LLM Reasoning
Sun Yat-Sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunieations（北京邮电大学）；Singapore Management University（新加坡管理大学）
Enhancing Depression Detection with Chain-of-Thought Prompting: From Emotion to Reasoning Using Large Language Models
College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学与工程学院）；Fujita Health University（藤田医科大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to i…
Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models
John F Kennedy High school Fremont, USA（约翰·F·肯尼迪高中弗里蒙特分校）；Queensland Academy for Science Mathematics and Technology（昆士兰科学数学与技术学院）；Research Spark Hub Inc（Research Spark Hub公司）
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）
Large language models can perform various reasoning tasks by using chain-of-thought prompting, which guides them to find answers through step-by-step demonstrations. However, the quality of the prompts depends on the demonstrations given to the models, and creating many of them by hand is costly. We introduce Synthetic prompting, a method that leverages a few handcrafted examples to prompt the model to generate more examples by itself, and selects effective demonstrations to elicit better reasoning. Our method alternates between a backward and forward process to generate new examples. The back…
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University（西湖大学工程学院）；City University of Hong Kong（香港城市大学）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；National University of Singapore（新加坡国立大学）；Peng Cheng Laboratory（鹏城实验室）
Chain-of-thought (CoT) prompting enables large language models (LLMs) to solve complex reasoning tasks by generating an explanation before the final prediction. Despite it's promising ability, a critical downside of CoT prompting is that the performance is greatly affected by the factuality of the generated explanation. To improve the correctness of the explanations, fine-tuning language models with explanation data is needed. However, there exists only a few datasets that can be used for such approaches, and no data collection tool for building them. Thus, we introduce CoTEVer, a tool-kit for…
Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Key Laboratory of Machine Perceptron (MOE), Peking University（北京大学机器感知教育部重点实验室）
Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University（新加坡管理大学）；Southwest Jiaotong University（西南交通大学）；Singapore University of Technology and Design（新加坡科技设计大学）；East China Normal University（华东师范大学）
