arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.00751cs.AIcs.LG
收藏

对齐再审视:大语言模型在陈述偏好与显示偏好上是否一致?

Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?

Department of Information Systems and Business Analytics(信息系统与商业分析系);University at Albany - State University of New York(阿尔巴尼大学 - 新 york 状态大学);Santa Clara, CA(圣克拉拉,加州);Princeton Junction, NJ(普林斯顿 junction,新泽西)

Zhuojun Gu, Quan Wang, Shuchu Han

AI总结:

本文提出测量大语言模型陈述偏好与情境化显示偏好之间偏差的方法,通过强制二元选择提示和KL散度量化,发现提示格式微小变化即可翻转选择,揭示LLM决策一致性与可控性不足的问题。

摘要:

大语言模型(LLMs)的最新进展凸显了使其行为与人类价值观保持一致的必要性。一个关键但尚未得到充分研究的问题是,LLM的陈述偏好(其报告的对一般原则的对齐)与其显示偏好(从情境化场景中的决策推断得出)之间可能存在偏差。这种偏差对LLM的可解释性、可信度、推理透明度以及伦理部署提出了根本性关切,尤其是在高风险应用中。本工作正式定义了这种偏好偏差,并提出了一种测量方法。我们研究了LLM如何在特定情境中激活不同的指导原则,从而导致其选择偏离先前陈述的一般原则。我们的方法包括精心构建一个由设计良好的提示组成的数据集,作为一系列强制二元选择,并将其呈现给LLM。我们将LLM对一般原则提示的响应(陈述偏好)与LLM对情境化提示的响应(显示偏好)进行比较,使用KL散度等指标来量化偏差。我们在不同偏好类别和四个主流LLM上重复了该分析,发现提示格式的微小变化往往会使首选选择发生翻转,无论测试中的偏好类别和LLM如何。这一普遍现象凸显了对LLM决策能力缺乏理解和控制。我们的研究对于将LLM整合到服务中至关重要,尤其是那些直接与人类交互的服务,其中道德、公平和社会责任是关键维度。此外,随着LLM越来越多地被设想用于自主智能体任务,在这些任务中,对所有LLM的中间决策步骤进行持续人类评估是不可能的,识别或意识到这种偏差将至关重要。

英文摘要:

Recent advances in Large Language Models (LLMs) highlight the need to align their behaviors with human values. A critical, yet understudied, issue is the potential divergence between an LLM's stated preferences (its reported alignment with general principles) and its revealed preferences (inferred from decisions in contextualized scenarios). Such deviations raise fundamental concerns for the interpretability, trustworthiness, reasoning transparency, and ethical deployment of LLMs, particularly in high-stakes applications. This work formally defines and proposes a method to measure this preference deviation. We investigate how LLMs may activate different guiding principles in specific contexts, leading to choices that diverge from previously stated general principles. Our approach involves crafting a rich dataset of well-designed prompts as a series of forced binary choices and presenting them to LLMs. We compare LLM responses to general principle prompts stated preference with LLM responses to contextualized prompts revealed preference, using metrics like KL divergence to quantify the deviation. We repeat the analysis across different categories of preferences and on four mainstream LLMs and find that a minor change in prompt format can often pivot the preferred choice regardless of the preference categories and LLMs in the test. This prevalent phenomenon highlights the lack of understanding and control of the LLM decision-making competence. Our study will be crucial for integrating LLMs into services, especially those that interact directly with humans, where morality, fairness, and social responsibilities are crucial dimensions. Furthermore, identifying or being aware of such deviation will be critically important as LLMs are increasingly envisioned for autonomous agentic tasks where continuous human evaluation of all LLMs' intermediary decision-making steps is impossible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑