FamiCom:进一步揭示语言模型提示的任务无关性能估计
FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation
AI总结:
本文提出FamiCom度量,将提示熟悉度与端任务复杂度结合,用于任务无关的性能估计,并在自动提示与示例选择中显著提升准确率。
摘要:
语言模型已展现出令人印象深刻的上下文学习能力,使其能够从输入提示中获益,并在下游端任务上取得更好表现。现有工作探究了这一现象背后的机制,并提出了与标签无关的提示度量,以更好地估计端任务性能。一种流行的方法是使用困惑度来衡量模型对提示的熟悉程度。尽管在域内任务上显示出一致的改进,但我们发现,困惑度等熟悉度度量无法在任务或领域迁移等复杂场景中准确估计性能。在本工作中,我们提出了一种修正的度量,称为FamiCom,为任务无关的性能估计提供了更全面的度量。具体而言,FamiCom将熟悉度与复杂度——即端任务的内在难度——相结合,而复杂度是当前度量中缺失的一个重要因素。实验表明,FamiCom与端任务性能强相关,产生了0.85的Spearman相关系数,而仅使用熟悉度的度量只有0.43。我们进一步将FamiCom应用于自动提示和示例选择,在准确率上比现有方法和基线高出7.0%以上。
英文摘要:
Language models have shown impressive in-context-learning capabilities, which allow them to benefit from input prompts and perform better on downstream end tasks. Existing works investigate the mechanisms behind this observation, and propose label-agnostic prompt metrics that can better estimate end-task performances. One popular approach is using perplexity as a way to measure models' familiarity with the prompt. While showing consistent improvements on in-domain tasks, we found that familiarity metrics such as perplexity cannot accurately estimate performance in complicated situations such as task or domain transferring scenarios. In this work, we propose a revised measure called FamiCom, providing a more comprehensive measure for task-agnostic performance estimation. Specifically, FamiCom combines familiarity with \textit{complexity} -- the inherent difficulty of end tasks, which is an important factor missing from current metrics. Experiments show that FamiCom strongly correlates with end-task performances, producing a 0.85 Spearman's correlation, versus 0.43 of familiarity-only ones'. We further apply FamiCom to automatic prompt and demonstration selection, and outperform existing methods and baselines by more than 7.0% in accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。