arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.09309cs.AI
收藏

重新缩放置信度:量表设计揭示LLM元认知

Rescaling confidence: What scale design reveals about LLM metacognition

INSAIT, Sofia University "St. Kliment Ohridski"(INSAIT,索菲亚大学‘圣克莱门特·欧里德斯基’)

Yuyang Dai, Yuxia Wang

AI总结:

研究LLM口头置信度受量表设计影响,发现0-20量表比标准0-100量表更有效提升元认知效率。

摘要:

口头置信度,即LLM报告数值确定性分数,被广泛用于估计黑箱环境中的不确定性,然而置信度量表本身(通常为0-100)很少被审视。我们表明,这一设计选择并非中性。在六个LLM和三个数据集上,口头置信度高度离散化,超过78%的响应集中在三个整数上。为研究这一现象,我们沿三个维度系统操纵置信度量表:粒度、边界位置和范围规律性,并使用$meta\ ext{-}d'$评估元认知敏感性。我们发现,0-20量表持续优于标准0-100格式,提高了元认知效率,而边界压缩会降低性能,即使在非规则范围下,整数偏好仍然存在。这些结果表明,置信度量表设计直接影响口头不确定性的质量,应被视为LLM评估中的首要实验变量。

英文摘要:

Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78\% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using $meta\text{-}d'$. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑