arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.09985cs.CLcs.AI
收藏

大型语言模型中的邓宁-克鲁格效应:对置信度校准的实证研究

The Dunning-Kruger Effect in Large Language Models: An Empirical Study of Confidence Calibration

Cognizant Technology Solutions(Cognizant技术解决方案)

Sudipta Ghosh, Mrityunjoy Panday

AI总结:

本研究发现大型语言模型在置信度校准上存在类似邓宁-克鲁格效应的过度自信现象,通过实验证明低性能模型更易高估自身能力。

摘要:

大型语言模型(LLMs)在多样任务上展现了显著的能力,但其准确评估自身置信度的能力仍不明确。我们进行了一项实证研究,调查LLMs是否表现出类似于邓宁-克鲁格效应的模式——一种认知偏差,即能力有限的人倾向于高估自己的能力。我们评估了四个最先进的模型(Claude Haiku 4.5、Gemini 2.5 Pro、Gemini 2.5 Flash和Kimi K2)在四个基准数据集上的表现,总计24,000次实验。我们的结果揭示了显著的校准差异:Kimi K2在仅23.3%准确率的情况下表现出严重的过度自信,预期校准误差(ECE)为0.726,而Claude Haiku 4.5达到最佳校准(ECE=0.122)时准确率为75.4%。这些发现表明,表现不佳的模型显示出明显更高的过度自信——一种类似于人类认知中的邓宁-克鲁格效应的模式。我们讨论了这对LLMs在高风险应用中安全部署的影响。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their ability to accurately assess their own confidence remains poorly understood. We present an empirical study investigating whether LLMs exhibit patterns reminiscent of the Dunning-Kruger effect -- a cognitive bias where individuals with limited competence tend to overestimate their abilities. We evaluate four state-of-the-art models (Claude Haiku 4.5, Gemini 2.5 Pro, Gemini 2.5 Flash, and Kimi K2) across four benchmark datasets totaling 24,000 experimental trials. Our results reveal striking calibration differences: Kimi K2 exhibits severe overconfidence with an Expected Calibration Error (ECE) of 0.726 despite only 23.3% accuracy, while Claude Haiku 4.5 achieves the best calibration (ECE = 0.122) with 75.4% accuracy. These findings demonstrate that poorly performing models display markedly higher overconfidence -- a pattern analogous to the Dunning-Kruger effect in human cognition. We discuss implications for safe deployment of LLMs in high-stakes applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑