ProSA：评估与理解 LLMs 的提示敏感性
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
AI总结：
本文提出 ProSA 框架，通过新指标 PromptSensiScore 和解码置信度评估并理解 LLMs 的提示敏感性，发现模型规模与少样本示例可提升鲁棒性，主观评估在复杂任务中更易受影响。
摘要：
大型语言模型（LLMs）在各类任务中展现了出色的能力，但其性能对所使用的提示高度敏感。这种变异性给准确评估和用户满意度带来了挑战。当前研究常常忽视实例级的提示变化及其对主观评估的影响。为弥补这些不足，我们提出了 ProSA，一个旨在评估和理解 LLMs 提示敏感性的框架。ProSA 引入了一种新颖的敏感度指标 PromptSensiScore，并利用解码置信度来阐明其潜在机制。我们跨越多项任务的广泛研究发现，提示敏感性在不同数据集和模型间存在波动，规模更大的模型表现出更强的鲁棒性。我们观察到，少样本示例可以缓解这一敏感性问题；同时，主观评估也易受提示敏感性的影响，尤其在复杂的、以推理为导向的任务中。此外，研究结果表明，模型置信度越高，提示鲁棒性越强。我们相信这项工作将成为研究 LLMs 提示敏感性的有用工具。项目发布于：https://github.com/open-compass/ProSA 。
英文摘要：
Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA, a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
