arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.03733cs.CLcs.AI
收藏

估计大型语言模型置信度的方法

Methods to Estimate Large Language Model Confidence

Duke University(杜克大学);Palo Alto Veterans Affairs Medical Center(帕洛阿尔托退伍军人事务医疗中心);Stanford University(斯坦福大学)

Maia Kotelanski, Robert Gallo, Ashwin Nayak, Thomas Savage

AI总结:

针对 GPT4 在复杂临床诊断中的不确定性表达问题,本文比较三类置信度估计方法,发现 SC Agreement Frequency 最能预测准确率。

摘要:

大型语言模型在传达不确定性方面存在困难,这是将 LLM 应用于复杂医疗任务的重大障碍。本研究评估了在针对具有挑战性的临床病例摘要提出诊断建议时衡量 LLM 置信度的方法。研究使用 Chain of Thought(思维链)和 Self Consistency(自一致性)提示向 GPT4 提出一系列具有挑战性的病例问题。研究考察了多种评估模型置信度的方法,并根据其预测模型所观察到准确率的能力进行评价。所评估的方法包括 Intrinsic Confidence(内在置信度)、SC Agreement Frequency(自一致性一致频率)和 CoT Response Length(思维链响应长度)。SC Agreement Frequency 与观察到的准确率相关,并且与 Intrinsic Confidence 和 CoT Length 分析相比,产生了更高的受试者工作特征曲线下面积。SC agreement 是模型置信度最有用的代理指标,尤其是在医疗诊断方面。Model Intrinsic Confidence 和 CoT Response Length 区分正确与错误答案的能力较弱,因此无法成为可靠且可解释的模型置信度标记。我们得出结论,GPT4 评估其自身诊断准确率的能力有限。SC Agreement Frequency 是衡量 GPT4 置信度最有用的方法。

英文摘要:

Large Language Models have difficulty communicating uncertainty, which is a significant obstacle to applying LLMs to complex medical tasks. This study evaluates methods to measure LLM confidence when suggesting a diagnosis for challenging clinical vignettes. GPT4 was asked a series of challenging case questions using Chain of Thought and Self Consistency prompting. Multiple methods were investigated to assess model confidence and evaluated on their ability to predict the models observed accuracy. The methods evaluated were Intrinsic Confidence, SC Agreement Frequency and CoT Response Length. SC Agreement Frequency correlated with observed accuracy, yielding a higher Area under the Receiver Operating Characteristic Curve compared to Intrinsic Confidence and CoT Length analysis. SC agreement is the most useful proxy for model confidence, especially for medical diagnosis. Model Intrinsic Confidence and CoT Response Length exhibit a weaker ability to differentiate between correct and incorrect answers, preventing them from being reliable and interpretable markers for model confidence. We conclude GPT4 has a limited ability to assess its own diagnostic accuracy. SC Agreement Frequency is the most useful method to measure GPT4 confidence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑