In recent years, large-scale language models (LLMs) have gained attention for their impressive text generation capabilities. However, these models often face the challenge of "hallucination," which undermines their reliability. In this study, we introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty. Human-defined methods for estimating uncertainty typically assume that "uncertainty is lower when the model's response is correct compared to when it is incorrect." However, setting a precise threshold to distinguish correctness is challenging. Therefore, we introduce uncertainty information as an intermediary variable that implicitly influences the model's behavior. Our innovative uncertainty-aware in-context learning framework involves fine-tuning the LLM using a calibration dataset. Our aim is to improve the model's responses by filtering out answers with high uncertainty while considering the model's knowledge limitations. We evaluate the model's knowledge by examining multiple responses to the same question for the presence of a correct answer. When the model lacks relevant knowledge, the response should indicate that the question cannot be answered. Conversely, when the model has relevant knowledge, the response should provide the correct answer. Extensive experiments confirm the effectiveness of our framework, leading to two key findings. First, the logit output values of the LLM partly reflect inherent uncertainty. Second, our model autonomously recognizes uncertainty, resulting in improved responses.
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);The University of Hong Kong(香港大学);The Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology, Guangzhou(香港科技大学(广州));MoE Key Laboratory of High Confidence Software Technologies(教育部高可信软件技术重点实验室)
ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
Applied AI Institute, Moscow, Russia(应用人工智能研究所,莫斯科,俄罗斯);SB AI Lab, Moscow, Russia(SB AI实验室,莫斯科,俄罗斯);Intellectual data analysis and predictive modeling Institute, Moscow, Russia(智能数据分析与预测建模研究所,莫斯科,俄罗斯)
EERPD: Leveraging Emotion and Emotion Regulation for Improving Personality Detection
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机学院多媒体信息处理国家重点实验室);School of Software, BNRist, Tsinghua University(清华大学软件学院北京信息科学与技术国家研究中心)
Improving Back-Translation with Uncertainty-based Confidence Estimation
ESTATES PTE LTD(六境私人有限公司);Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing Advanced Innovation Center for Language Resources(北京语言资源高精尖创新中心)
Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires
University of Amsterdam(阿姆斯特丹大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Johns Hopkins University(约翰斯·霍普金斯大学);National Institutes of Health(美国国立卫生研究院);Allen Institute for AI(艾伦人工智能研究所)
Leveraging Historical Interaction Data for Improving Conversational Recommender System
School of Information, Renmin University of China(中国人民大学信息学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室);Meituan-Dianping Group(美团点评集团)
Improving Stance Detection by Leveraging Measurement Knowledge from Social Sciences: A Case Study of Dutch Political Tweets and Traditional Gender Role Division
Department of Methodology and Statistics(方法学与统计学系)
Improving Background Based Conversation with Context-aware Knowledge Pre-selection
University of Amsterdam(阿姆斯特丹大学)
基于背景的对话(Background Based Conversations,BBCs)已经被开发以通过利用背景知识来使对话系统生成更多信息且更自然的响应。现有的BBC方法可以分为两类:基于提取的方法和基于生成的方法。前一个摘录来自背景材料,作为不一定自然的反应。后者产生的反应是自然的,但不一定有效地利用背景知识。在本文中,我们专注于基于生成的方法,并提出了一个模型,即上下文感知的知识预选择(CaKe),它引入了一个预选择过程,使用动态的双向注意,以改善知识的选择,使用的话语历史上下文作为先验信息,以选择最相关的背景材料。实验结果表明,我们的模型优于当前最先进的基线,表明它受益于预选过程,从而提高信息量和流畅性。
Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning
National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Tencent Inc.(腾讯公司);Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University(江苏省语言能力协同创新中心,江苏师范大学)