更多的RLHF意味着更多的信任？关于偏好对齐对可信度的影响
More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness
AI总结：
本研究调查了RLHF对LLM在五个可信度维度的影响，发现其不能自动保证可信度，并提出了基于影响函数的数据归因方法以理解微调数据的作用，强调需从数据和框架角度改进模型对齐。
摘要：
大型语言模型（LLMs）的可信度是指其输出可靠、安全和符合伦理的程度，它已成为与认知性能并列的关键考量因素。在实践中，基于人类反馈的强化学习（RLHF）被广泛用于将LLMs与标注的人类偏好对齐，但其对模型可信度的假定影响尚未得到严格评估。为了填补这一知识空白，本研究调查了使用通用偏好数据对齐的模型在五个可信度垂直领域中的表现：毒性、刻板偏见、机器伦理、真实性和隐私。我们的结果表明，基于人类偏好的RLHF并不能自动保证可信度，且经常观察到相反的效果。此外，我们提出将基于高效影响函数的数据归因方法适配到RLHF设置中，以更好地理解微调数据对各个可信度基准的影响，并通过提供我们估计的归因分数来展示其可行性。总之，我们的结果强调需要从数据和框架两个角度采用更细致的方法进行模型对齐，我们希望这项研究能引导社区开发出在不牺牲可信度的情况下日益强大的语言模型。
英文摘要：
The trustworthiness of Large Language Models (LLMs) refers to the extent to which their outputs are reliable, safe, and ethically aligned, and it has become a crucial consideration alongside their cognitive performance. In practice, Reinforcement Learning From Human Feedback (RLHF) has been widely used to align LLMs with labeled human preferences, but its assumed effect on model trustworthiness hasn't been rigorously evaluated. To bridge this knowledge gap, this study investigates how models aligned with general-purpose preference data perform across five trustworthiness verticals: toxicity, stereotypical bias, machine ethics, truthfulness, and privacy. Our results demonstrate that RLHF on human preferences doesn't automatically guarantee trustworthiness, and reverse effects are often observed. Furthermore, we propose to adapt efficient influence function based data attribution methods to the RLHF setting to better understand the influence of fine-tuning data on individual trustworthiness benchmarks, and show its feasibility by providing our estimated attribution scores. Together, our results underscore the need for more nuanced approaches for model alignment from both the data and framework perspectives, and we hope this research will guide the community towards developing language models that are increasingly capable without sacrificing trustworthiness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
