在语言模型不确定性估计中考虑谄媚行为
Accounting for Sycophancy in Language Model Uncertainty Estimation
AI总结：
本文首次研究语言模型中谄媚偏差与不确定性估计的关系，提出推广的谄媚偏差定义和新算法SyRoUP，实验证明用户置信度关键调节谄媚影响，且SyRoUP能更好预测这些影响。
摘要：
有效的人机协作要求机器学习模型将不确定性外部化，以便用户在必要时进行反思和干预。对于语言模型而言，这些不确定性表示可能受到谄媚偏差的影响：即倾向于同意用户，即使用户是错误的。例如，模型可能对用户建议的（错误）问题解决方案过度自信。我们首次研究了谄媚行为与不确定性估计之间的关系。我们提出了谄媚偏差定义的推广形式，以衡量其对不确定性估计的下游影响，并提出了一种新算法（SyRoUP），用于在不确定性估计过程中考虑谄媚行为。与以往关于谄媚行为的研究不同，我们研究了广泛的用户行为，同时改变用户建议的正确性和置信度，以观察模型答案（及其确定性）如何变化。我们在对话预测和问答任务上的实验表明，用户置信度在调节谄媚行为的影响中起着关键作用，并且SyRoUP能够更好地预测这些影响。基于这些结果，我们认为将模型和用户的不确定性同时外部化有助于减轻谄媚偏差的影响。
英文摘要：
Effective human-machine collaboration requires machine learning models to externalize uncertainty, so users can reflect and intervene when necessary. For language models, these representations of uncertainty may be impacted by sycophancy bias: proclivity to agree with users, even if they are wrong. For instance, models may be over-confident in (incorrect) problem solutions suggested by a user. We study the relationship between sycophancy and uncertainty estimation for the first time. We propose a generalization of the definition of sycophancy bias to measure downstream impacts on uncertainty estimation, and also propose a new algorithm (SyRoUP) to account for sycophancy in the uncertainty estimation process. Unlike previous works on sycophancy, we study a broad array of user behaviors, varying both correctness and confidence of user suggestions to see how model answers (and their certainty) change. Our experiments across conversation forecasting and question-answering tasks show that user confidence plays a critical role in modulating the effects of sycophancy, and that SyRoUP can better predict these effects. From these results, we argue that externalizing both model and user uncertainty can help to mitigate the impacts of sycophancy bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
