大语言模型提示工程领域的已有研究提出了多种无梯度的基于概率的提示选择方法,旨在为给定任务从候选提示中选出最优提示,但这些方法之间缺乏全面且公平的比较。本文提出一个统一框架,用于解读和评估现有的基于概率的提示选择方法,并在13个常见且多样化的NLP任务上开展了大量实验。研究发现,所有现有方法都可被解读为最大化输入与预测输出之间互信息(MI)方法的某种变体。基于这一发现,我们开发了MI的其他几种组合变体,将预言机提示选择方法的有效性从87.79%提升至94.98%(该有效性以所选提示的性能与最优预言机提示的性能之比衡量)。此外,考虑到所有方法都依赖可能存在偏差的模型输出概率分布,我们提出了一种名为边际化校准(Calibration by Marginalization, CBM)的新型校准方法,该方法与现有方法正交,可将最佳方法的提示选择有效性提升至96.85%,在无校准情况下达到预言机提示F1值的99.44%。
英文摘要:
Previous works in prompt engineering for large language models have introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.
A Toolbox for Improving Evolutionary Prompt Search
Institute for Applied Artificial Intelligence (IAAI), Stuttgart Media University(应用人工智能研究所(IAAI)、斯图加特媒体大学);Institute for Natural Language Processing (IMS), University of Stuttgart(自然语言处理研究所(IMS)、斯图加特大学)
Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization
Centre for Software Excellence, Huawei, Canada(华为加拿大软件卓越中心);Department of Computer Science, University of Manitoba, Canada(曼尼托巴大学计算机科学系);School of Computing, Queen’s University, Canada(皇后大学计算学院)
Improving Task Generalization via Unified Schema Prompt
Sun Yat-sen University(中山大学);Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Langboat Technology(澜舟科技);Microsoft Research Asia(微软亚洲研究院)
When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation
Charles Sturt University(查尔斯斯特大学);Chittagong University of Engineering and Technology(吉大港工程技术大学);Bangladesh Army University of Science and Technology(孟加拉国陆军理工大学)
PACE: Improving Prompt with Actor-Critic Editing for Large Language Model
Key Laboratory of High Confidence Software Technologies (Peking University)(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院)
Dr.Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian
National University of Science and Technology POLITEHNICA Bucharest(波兰技术科学大学);Dalle Molle Institute for Artificial Intelligence Research (IDSIA)(达勒莫利人工智能研究所);MedicChat
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Singapore Management University(新加坡管理大学);Southwest Jiaotong University(西南交通大学);Singapore University of Technology and Design(新加坡科技设计大学);East China Normal University(华东师范大学)
Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
Université Paris 1 Panthéon-Sorbonne(巴黎第一大学先贤祠-索邦大学);Université Paris-Saclay(巴黎萨克雷大学);Institut Jean Nicod(让·尼科德研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL-EHESS-CNRS(巴黎文理研究大学-法国高等社会科学研究院-法国国家科学研究中心)
SPRIG: Improving Large Language Model Performance by System Prompt Optimization
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Michigan(密歇根大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);LG AI Research(LG AI 研究院)
Improving Complex Reasoning with Dynamic Prompt Corruption: A soft prompt Optimization Approach
Zhejiang University(浙江大学);Hangzhou YunQi Academy of Engineering(杭州云栖工程学院);College of Computer Science and Technology, Zhejiang University of Technology(浙江工业大学计算机科学与技术学院);Alibaba Cloud Computing(阿里巴巴云计算)
Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs
Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia(计算机系统部,乔塞夫·斯塔芬研究所,卢布尔雅那,斯洛文尼亚);Jožef Stefan International Postgraduate School, Ljubljana, Slovenia(乔塞夫·斯塔芬国际研究生学院,卢布尔雅那,斯洛文尼亚);Center for Astrophysics and Cosmology, University of Nova Gorica, Nova Gorica, Slovenia(天体物理与宇宙学中心,诺瓦戈里察大学,诺瓦戈里察,斯洛文尼亚)