arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.02185cs.CLcs.AIcs.LG
收藏

POSIX:一种用于大语言模型的提示敏感性指数

POSIX: A Prompt Sensitivity Index For Large Language Models

Indian Institute of Technology Delhi(印度理工学院德里分校);Adobe Inc.(奥多比公司)

Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, Tanmoy Chakraborty

AI总结:

本文提出提示敏感性指数POSIX,通过衡量替换意图不变提示时响应对数似然的相对变化来评估大语言模型的提示敏感性,并发现增加参数或指令微调未必降低敏感性,而加入few-shot示例通常能显著降低敏感性。

摘要:

尽管大语言模型(LLMs)具有卓越的能力,但研究发现它们对提示中的微小变化出奇地敏感,往往会对提示中的微小变化(如拼写错误、措辞改变或提示模板变化)产生显著不同的输出。然而,在评估大语言模型的质量时,关注点往往只集中在其下游任务的表现上,而对提示敏感性几乎或完全没有关注。为填补这一空白,我们提出了 POSIX——一种新颖的提示敏感性指数(PrOmpt Sensitivity IndeX),作为衡量提示敏感性的可靠指标,从而提供对大语言模型性能更全面的评估。POSIX 的核心思想是:在将相应提示替换为另一个保持意图不变的提示时,捕捉给定响应对数似然的相对变化。我们提供了充分的实证证据,证明 POSIX 在捕捉提示敏感性方面的有效性,并随后使用它来测量并比较各种开源大语言模型的提示敏感性。我们发现,仅仅增加参数数量或进行指令微调并不一定能降低提示敏感性,而加入少量 few-shot 示例(即使只有一个)几乎总能显著降低提示敏感性。我们还发现,在多选题(MCQ)类型任务中,提示模板的改变会导致最高的敏感性,而在开放式生成任务中,改写(paraphrasing)会导致最高的敏感性。用于复现我们结果的代码已在 https://github.com/kowndinya-renduchintala/POSIX 开源。

英文摘要:

Despite their remarkable capabilities, Large Language Models (LLMs) are found to be surprisingly sensitive to minor variations in prompts, often generating significantly divergent outputs in response to minor variations in the prompts, such as spelling errors, alteration of wording or the prompt template. However, while assessing the quality of an LLM, the focus often tends to be solely on its performance on downstream tasks, while very little to no attention is paid to prompt sensitivity. To fill this gap, we propose POSIX - a novel PrOmpt Sensitivity IndeX as a reliable measure of prompt sensitivity, thereby offering a more comprehensive evaluation of LLM performance. The key idea behind POSIX is to capture the relative change in loglikelihood of a given response upon replacing the corresponding prompt with a different intent-preserving prompt. We provide thorough empirical evidence demonstrating the efficacy of POSIX in capturing prompt sensitivity and subsequently use it to measure and thereby compare prompt sensitivity of various open-source LLMs. We find that merely increasing the parameter count or instruction tuning does not necessarily reduce prompt sensitivity whereas adding some few-shot exemplars, even just one, almost always leads to significant decrease in prompt sensitivity. We also find that alterations to prompt template lead to the highest sensitivity in the case of MCQ type tasks, whereas paraphrasing results in the highest sensitivity in open-ended generation tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinya-renduchintala/POSIX.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑