arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.21776cs.CL
收藏

PromptNCE:仅使用LLM和对比估计提示进行点互信息预测

PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts

Department of Computer Science Stanford University(计算机科学系 斯坦福大学)

Juliette Woodrow, Chris Piech

AI总结:

本文提出PromptNCE方法,通过将条件概率估计转化为对比任务,并引入显式的OTHER类别来恢复真实的条件概率,从而在低数据情况下实现零样本点互信息估计。

摘要:

从文本中估计互信息通常需要训练特定任务的批评者,这限制了其在低数据设置中的应用。我们问大语言模型能否仅通过提示和提取的概率来零样本估计点互信息。我们引入了一个包含三个公开可用数据集的人类衍生真实PMI基准,并评估了五个信息论提示基于的估计器。我们的主要方法PromptNCE将条件概率估计框架为对比任务,并在候选集上引入显式的OTHER类别。我们理论证明,添加OTHER类别可以恢复真实的条件P(y | x),而不是仅仅在列出的候选者之间进行排名,将对比提示转化为通用的零样本概率估计器。PromptNCE在所有三个数据集上都是最佳的零样本方法,与人类衍生的PMI达到Spearman相关性高达0.82。我们还展示了在计算机科学教育中的案例研究,说明这些估计器如何在低数据情况下用于评分学生知识摘要。

英文摘要:

Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings. We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities. We construct a benchmark from three publicly available human-annotated datasets with ground-truth PMI, and evaluate five information-theoretic prompting-based estimators. Our main method, PromptNCE, frames conditional probability estimation as a contrastive task and augments the candidate set with an explicit OTHER category. The OTHER category allows the model to assign probability mass outside the candidate set, avoiding the closed-set normalization of standard contrastive prompts. PromptNCE gives the best conditional probability estimates on all three datasets. For full PMI, we find that estimating label base rates is the primary bottleneck on two of the three datasets, with the best methods reaching Spearman correlation up to 0.78. We also present a case study in computer science education showing how these estimators can be used to score student knowledge summaries in a low-data setting. We release our code and prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑