arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14877cs.CL
收藏

通过统一评估与分析改进基于概率的提示选择

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

University of Washington(华盛顿大学);KAIST(韩国科学技术院);UCL(伦敦大学学院);Dongguk University(东国大学)

Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, Minjoon Seo

AI总结:

针对大语言模型基于概率的无梯度提示选择方法缺乏公平比较的问题,本文提出统一评估框架,发现现有方法均为互信息最大化的变体,通过组合变体及新型边际化校准方法显著提升了提示选择有效性。

摘要:

大语言模型提示工程领域的已有研究提出了多种无梯度的基于概率的提示选择方法,旨在为给定任务从候选提示中选出最优提示,但这些方法之间缺乏全面且公平的比较。本文提出一个统一框架,用于解读和评估现有的基于概率的提示选择方法,并在13个常见且多样化的NLP任务上开展了大量实验。研究发现,所有现有方法都可被解读为最大化输入与预测输出之间互信息(MI)方法的某种变体。基于这一发现,我们开发了MI的其他几种组合变体,将预言机提示选择方法的有效性从87.79%提升至94.98%(该有效性以所选提示的性能与最优预言机提示的性能之比衡量)。此外,考虑到所有方法都依赖可能存在偏差的模型输出概率分布,我们提出了一种名为边际化校准(Calibration by Marginalization, CBM)的新型校准方法,该方法与现有方法正交,可将最佳方法的提示选择有效性提升至96.85%,在无校准情况下达到预言机提示F1值的99.44%。

英文摘要:

Previous works in prompt engineering for large language models have introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑