We study the task of prompting large-scale language models to perform multi-step reasoning. Existing work shows that when prompted with a chain of thoughts (CoT), sequences of short sentences describing intermediate reasoning steps towards a final answer, large language models can generate new reasoning chains and predict answers for new inputs. A central question is which reasoning examples make the most effective prompts. In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning. We show that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, achieve substantially better performance on multi-step reasoning tasks over strong baselines. We further extend our complexity-based criteria from prompting (selecting inputs) to decoding (selecting outputs), where we sample multiple reasoning chains from the model, then choose the majority of generated answers from complex reasoning chains (over simple chains). When used to prompt GPT-3 and Codex, our approach substantially improves multi-step reasoning accuracy and achieves new state-of-the-art (SOTA) performance on three math benchmarks (GSM8K, MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared with existing example selection schemes like manual tuning or retrieval-based selection, selection based on reasoning complexity is intuitive, easy to implement, and annotation-efficient. Further results demonstrate the robustness of performance gains from complex prompts under format perturbation and distribution shift.
Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models
Shanghai Jiao Tong University（上海交通大学）；MoE Key Lab of Artificial Intelligence（教育部人工智能重点实验室）；AI Institute（人工智能研究院）；State Key Lab of Advanced Optical Communication System and Network（先进光通信系统与网络国家重点实验室）
Diverge to Induce Prompting: Multi-Rationale Induction for Zero-Shot Reasoning
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan（国家台湾大学计算机科学与信息工程系）；Institute of Information Science, Academia Sinica, Taiwan（学术院信息科学研究所）；AI Research Center (AINTU), National Taiwan University, Taiwan（国家台湾大学人工智能研究中心（AINTU））
PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoning
The College of Artificial Intelligence, Nanjing University of Aeronautics（人工智能学院，南京航空航天大学）；Astronautics The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（航空航天脑机智能技术重点实验室，教育部）
Advancing Multi-Step Mathematical Reasoning in Large Language Models through Multi-Layered Self-Reflection with Auto-Prompting
Luiz de Queiroz College of Agriculture University of São Paulo（圣保罗大学农业学院）；School of Engineering and Sciences Tecnologico de Monterrey（蒙特雷技术学院工程与科学学院）；Institute of Mathematics and Computer Sciences University of São Paulo（圣保罗大学数学与计算机科学研究所）
DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoning
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Computer and Cyber Sciences, Augusta University（奥古斯塔大学计算机与网络科学学院）
Hint of Pseudo Code (HoPC): Zero-Shot Step by Step Pseudo Code Reasoning Prompting
Department of Computer Science and Technology（计算机科学与技术系）；Institute for Artificial Intelligence at Tsinghua University（清华大学人工智能研究院）；Beijing National Research Center for Information Science and Technology（北京信息科学国家研究中心）；Tsinghua University（清华大学）
CoEvoT: Co-Evolving Chain-of-Thought Prompting for Graph-LLM Reasoning
Sun Yat-Sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunieations（北京邮电大学）；Singapore Management University（新加坡管理大学）
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University（西湖大学工程学院）；City University of Hong Kong（香港城市大学）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；Johns Hopkins University（约翰斯·霍普金斯大学）
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University（新加坡管理大学）；Southwest Jiaotong University（西南交通大学）；Singapore University of Technology and Design（新加坡科技设计大学）；East China Normal University（华东师范大学）
Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Prompting
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（电子工程学院，上海工程技术大学）；School of Information Science and Engineering, Ningbo University（信息科学与工程学院，宁波大学）
PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models
Institute of Data Science（数据科学研究所）；NUS Centre for Trusted Internet and Community, National University of Singapore（新加坡国立大学可信互联网与社区中心）；Amazon（亚马逊）；Stanford University（斯坦福大学）；Tsinghua University（清华大学）
Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Key Laboratory of Machine Perceptron (MOE), Peking University（北京大学机器感知教育部重点实验室）
