arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.04037cs.CL
收藏

通过困惑度估计揭示语言模型中的提示机制

Demystifying Prompts in Language Models via Perplexity Estimation

Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Meta AI Research(元人工智能研究部);Allen Institute for Artificial Intelligence(艾伦人工智能研究所)

Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer

AI总结:

该研究针对语言模型提示选择影响性能的问题,提出提示性能与模型对其语言熟悉度相关的假设,验证了低困惑度提示效果更优,并设计了基于释义扩展和困惑度筛选的提示生成方法以提升任务性能。

摘要:

语言模型可通过提示执行多种零样本和少样本学习任务。然而,性能会因提示的选择而出现显著差异,我们目前尚不理解这种现象的成因,也不清楚如何挑选最优提示。在本研究中,我们分析了导致这种差异的因素,并提出了一项新的经验假设:提示的性能与模型对其所含语言的熟悉程度相关。在广泛的任务中,我们发现提示的困惑度越低,其完成任务的效果越好。基于此,我们设计了一种提示生成方法:(1)利用GPT3和回译进行释义,自动扩展少量人工编写的种子提示集;(2)选择困惑度最低的提示,以实现性能的显著提升。

英文摘要:

Language models can be prompted to perform a wide variety of zero- and few-shot learning problems. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens or how to pick the best prompts. In this work, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task. As a result, we devise a method for creating prompts: (1) automatically extend a small seed set of manually written prompts by paraphrasing using GPT3 and backtranslation and (2) choose the lowest perplexity prompts to get significant gains in performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑