arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11278cs.CL
收藏

不要设计,要学习:用于生成式大语言模型不确定性估计的可训练评分函数

Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs

University of Southern California(南加州大学);University of Birmingham(伯明翰大学);Amazon AI(亚马逊人工智能)

Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, Salman Avestimehr

AI总结:

针对生成式大语言模型不确定性估计中现有基于概率的评分函数存在的局限,提出可学习响应评分(LARS)函数,通过监督数据捕捉token与概率的复杂依赖,实验显示其在多任务多数据集上AUROC分数提升高达16%。

摘要:

生成式大语言模型(LLMs)的不确定性估计(UE)对于评估生成序列的可靠性至关重要。UE方法的一个重要子集利用token概率来评估不确定性,通过评分函数将多个token概率聚合为单个UE分数。现有的基于概率的UE评分函数(如长度归一化评分和基于语义贡献的加权)旨在解决问题的某些方面,但存在局限性,包括无法处理有偏概率以及token之间复杂的语义依赖关系。为解决这些问题,本工作提出可学习响应评分(LARS)函数,这是一种新型评分函数,利用监督数据捕捉token与概率之间的复杂依赖关系,从而在计算LLM生成结果的不确定性时产生更可靠、更校准的响应分数。我们在问答和算术推理任务中使用多种数据集进行的综合实验表明,LARS显著优于现有评分函数,AUROC分数提升高达16%。

英文摘要:

Uncertainty estimation (UE) of generative large language models (LLMs) is crucial for evaluating the reliability of generated sequences. A significant subset of UE methods utilize token probabilities to assess uncertainty, aggregating multiple token probabilities into a single UE score using a scoring function. Existing scoring functions for probability-based UE, such as length-normalized scoring and semantic contribution-based weighting, are designed to solve certain aspects of the problem but exhibit limitations, including the inability to handle biased probabilities and complex semantic dependencies between tokens. To address these issues, in this work, we propose Learnable Response Scoring (LARS) function, a novel scoring function that leverages supervised data to capture complex dependencies between tokens and probabilities, thereby producing more reliable and calibrated response scores in computing the uncertainty of LLM generations. Our comprehensive experiments across question-answering and arithmetical reasoning tasks with various datasets demonstrate that LARS significantly outperforms existing scoring functions, achieving improvements of up to 16\% AUROC score.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑