arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10189cs.CLcs.LG
收藏

大型语言模型上下文学习的不确定性量化

Uncertainty Quantification for In-Context Learning of Large Language Models

Emory University(埃默里大学);NEC Labs America(美国NEC实验室);Microsoft(微软公司);NEC Corporation(日本电气公司)

Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen

AI总结:

本文针对大型语言模型上下文学习中的预测不确定性,提出一种即插即用的无监督估计方法,将不确定性分解为演示引起的偶然不确定性和模型配置引起的认知不确定性,并通过大量实验验证其有效性。

摘要:

上下文学习已成为大型语言模型(LLMs)的一项突破性能力,通过在提示中提供少量与任务相关的演示,彻底改变了众多领域。然而,LLM响应的可信性问题(如幻觉)也引发了广泛讨论。现有工作致力于量化LLM响应的不确定性,但往往忽视了LLM的复杂本质以及上下文学习的独特性。在本工作中,我们深入研究了与上下文学习相关的LLM预测不确定性,强调此类不确定性可能同时来源于所提供的演示(偶然不确定性)以及与模型配置相关的模糊性(认知不确定性)。我们提出了一种新颖的公式化表述及相应的估计方法,用于量化这两类不确定性。所提方法以即插即用的方式提供了一种无监督途径来理解上下文学习的预测。大量实验验证了该分解方法的有效性。代码和数据可在以下地址获取:https://github.com/lingchen0331/UQ_ICL。

英文摘要:

In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) and revolutionized various fields by providing a few task-relevant demonstrations in the prompt. However, trustworthy issues with LLM's response, such as hallucination, have also been actively discussed. Existing works have been devoted to quantifying the uncertainty in LLM's response, but they often overlook the complex nature of LLMs and the uniqueness of in-context learning. In this work, we delve into the predictive uncertainty of LLMs associated with in-context learning, highlighting that such uncertainties may stem from both the provided demonstrations (aleatoric uncertainty) and ambiguities tied to the model's configurations (epistemic uncertainty). We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. Extensive experiments are conducted to demonstrate the effectiveness of the decomposition. The code and data are available at: https://github.com/lingchen0331/UQ_ICL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑