UAlign：利用不确定性估计实现大型语言模型的事实一致性对齐
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
AI总结：
UAlign框架利用不确定性估计表示知识边界，并将其作为提示特征，通过PPO算法对齐LLM事实知识，显著提升模型自信回答已知问题与拒绝未知问题的能力。
摘要：
尽管大型语言模型（LLMs）展示了令人印象深刻的能力，但它们仍然经常难以准确表达其所拥有的事实知识，尤其是在LLMs的知识边界模糊的情况下。为了提高LLMs的事实表达准确性，我们提出了UAlign框架，该框架利用不确定性估计来表示知识边界，然后将这些表示作为输入特征显式地纳入提示中，使LLMs与事实知识对齐。首先，我们通过计算两种不确定性估计（包括置信度分数和语义熵）来准备知识问答（QA）样本的数据集，以表示LLMs的知识边界。随后，利用准备好的数据集，我们训练了一个结合不确定性估计的奖励模型，然后采用近端策略优化（PPO）算法对LLMs进行事实一致性对齐。实验结果表明，通过在LLM对齐中整合不确定性表示，所提出的UAlign能够显著增强LLMs在领域内和领域外任务中自信地回答已知问题和拒绝未知问题的能力，显示出相对于各种基于提示和训练基线的可靠性提升和良好的泛化性。
英文摘要：
Despite demonstrating impressive capabilities, Large Language Models (LLMs) still often struggle to accurately express the factual knowledge they possess, especially in cases where the LLMs' knowledge boundaries are ambiguous. To improve LLMs' factual expressions, we propose the UAlign framework, which leverages Uncertainty estimations to represent knowledge boundaries, and then explicitly incorporates these representations as input features into prompts for LLMs to Align with factual knowledge. First, we prepare the dataset on knowledge question-answering (QA) samples by calculating two uncertainty estimations, including confidence score and semantic entropy, to represent the knowledge boundaries for LLMs. Subsequently, using the prepared dataset, we train a reward model that incorporates uncertainty estimations and then employ the Proximal Policy Optimization (PPO) algorithm for factuality alignment on LLMs. Experimental results indicate that, by integrating uncertainty representations in LLM alignment, the proposed UAlign can significantly enhance the LLMs' capacities to confidently answer known questions and refuse unknown questions on both in-domain and out-of-domain tasks, showing reliability improvements and good generalizability over various prompt- and training-based baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
