arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02451cs.AI
收藏

强偏好影响偏好模型与价值对齐的鲁棒性

Strong Preferences Affect the Robustness of Preference Models and Value Alignment

National University of Singapore(新加坡国立大学)

Ziwei Xu, Mohan Kankanhalli

AI总结:

本文通过理论分析Bradley-Terry和Placket-Luce偏好模型的敏感性,揭示强偏好(概率接近0或1)会显著影响其他偏好预测,从而影响价值对齐的鲁棒性与安全性。

摘要:

价值对齐旨在确保大语言模型(LLMs)及其他AI智能体按照人类价值观行事,这对于保障这些系统的安全性和可信度至关重要。价值对齐的一个关键组成部分是将人类偏好建模为人类价值观的表示。本文通过考察偏好模型的敏感性来研究价值对齐的鲁棒性。具体而言,我们提出如下问题:某些偏好概率的变化如何影响这些模型对其他偏好的预测?为回答该问题,我们通过分析广泛使用的偏好模型对其所建模偏好的微小变化的敏感性,从理论上分析其鲁棒性。研究发现,在Bradley-Terry模型和Placket-Luce模型中,一个偏好的概率会随着其他偏好的变化而发生显著变化,尤其是当这些偏好处于主导地位(即概率接近0或1)时。我们确定了这些模型敏感性变得显著的具体条件,并讨论了其对AI系统中价值对齐的鲁棒性与安全性的实际影响。

英文摘要:

Value alignment, which aims to ensure that large language models (LLMs) and other AI agents behave in accordance with human values, is critical for ensuring safety and trustworthiness of these systems. A key component of value alignment is the modeling of human preferences as a representation of human values. In this paper, we investigate the robustness of value alignment by examining the sensitivity of preference models. Specifically, we ask: how do changes in the probabilities of some preferences affect the predictions of these models for other preferences? To answer this question, we theoretically analyze the robustness of widely used preference models by examining their sensitivities to minor changes in preferences they model. Our findings reveal that, in the Bradley-Terry and the Placket-Luce model, the probability of a preference can change significantly as other preferences change, especially when these preferences are dominant (i.e., with probabilities near 0 or 1). We identify specific conditions where this sensitivity becomes significant for these models and discuss the practical implications for the robustness and safety of value alignment in AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑