Large language models (LLMs) have been found to produce hallucinations when the question exceeds their internal knowledge boundaries. A reliable model should have a clear perception of its knowledge boundaries, providing correct answers within its scope and refusing to answer when it lacks knowledge. Existing research on LLMs' perception of their knowledge boundaries typically uses either the probability of the generated tokens or the verbalized confidence as the model's confidence in its response. However, these studies overlook the differences and connections between the two. In this paper, we conduct a comprehensive analysis and comparison of LLMs' probabilistic perception and verbalized perception of their factual knowledge boundaries. First, we investigate the pros and cons of these two perceptions. Then, we study how they change under questions of varying frequencies. Finally, we measure the correlation between LLMs' probabilistic confidence and verbalized confidence. Experimental results show that 1) LLMs' probabilistic perception is generally more accurate than verbalized perception but requires an in-domain validation set to adjust the confidence threshold. 2) Both perceptions perform better on less frequent questions. 3) It is challenging for LLMs to accurately express their internal confidence in natural language.
Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA(加州大学圣地亚哥分校计算机科学与工程系,拉贾尔,加利福尼亚州,美国);Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, California, USA(加州大学圣地亚哥分校Halıcıoğlu数据科学研究所,拉贾尔,加利福尼亚州,美国);Department of Statistics, Stanford University, Stanford, California, USA(斯坦福大学统计学系,斯坦福,加利福尼亚州,美国)
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
University of Amsterdam(阿姆斯特丹大学);University of Groningen(格罗宁根大学);University of Edinburgh(爱丁堡大学);IT University of Copenhagen(哥本哈根信息技术大学);Pioneer Centre for Artificial Intelligence(先锋人工智能中心)
Can LLM Rerankers Predict Their Own Ranking Performance?
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs
Faculty of Computer Science, University of Vienna, Vienna, Austria(维也纳大学计算机科学系);UniVie Doctoral School Computer Science, Vienna, Austria(UniVie计算机科学博士学院);Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria(维也纳大学语言与文化研究系)
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
Tsinghua University, Institute for Interdisciplinary Information Sciences(清华大学,交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院);South China Normal University, School of Electronic Science and Engineering(华南师范大学,电子科学与工程学院);Guangzhou Richstone Data Technologies Co., Ltd.(广州瑞驰数据技术有限公司);LMU Munich & Munich Center for Machine Learning(慕尼黑大学 & 慕尼黑机器学习中心)
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
National University of Singapore(新加坡国立大学);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);The Hong Kong University of Science and Technology(香港科技大学)
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles
Faculty of Computer Science, University of Vienna(维也纳大学计算机学院);UniVie Doctoral School Computer Science(维也纳大学计算机博士学院);Department of Computer Science, Aarhus University(奥胡斯大学计算机系);Faculty of Philological and Cultural Studies, University of Vienna(维也纳大学语文学与文化学院)