arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12134cs.AIcs.HC
收藏

价值对齐税:在大语言模型对齐中测量价值权衡

Value Alignment Tax: Measuring Value Trade-offs in LLM Alignment

Center for Data Science, NYU Shanghai(纽约大学上海分校数据科学中心)

Jiajun Chen, Hua Shen

AI总结:

本文提出VAT框架,用于测量对齐干预如何影响价值系统,揭示目标值与非目标值之间的系统性权衡,通过实验数据展示对齐过程中的潜在风险。

摘要:

现有价值对齐研究通常静态地刻画价值关系,忽视了对齐干预如提示、微调或偏好优化如何重塑更广泛的价值体系。在实践中,对齐目标价值可能隐式地转移其他价值,产生未被测量的价值权衡。我们引入VAT框架,通过测量对齐引起的改变如何在相互关联的价值之间传播,以量化价值权衡。VAT捕捉了在对齐干预下价值表达的系统动态,使评估预期改进和意外副作用成为可能。使用基于Schwartz价值理论的受控场景-动作数据集,我们收集了成对的预后规范判断,并分析了不同模型、价值和干预的对齐效果。结果表明,对齐往往产生不均且结构化的价值共动,揭示了目标与非目标价值之间的系统性权衡。这些影响在传统目标-only评估下往往不可见,但通过VAT变得明显,突显了过程层面的对齐风险,并提供了关于LLM中价值对齐动态性质的新见解。数据集和代码已开源。

英文摘要:

Existing work on value alignment typically characterizes value relations statically, ignoring how alignment interventions, such as prompting, fine-tuning, or preference optimization, reshape the broader value system. In practice, aligning a target value can implicitly shift other values, creating value trade-offs that remain largely unmeasured. We introduce VAT, a framework that quantifies value trade-offs by measuring how alignment-induced changes propagate across interconnected values relative to achieved on-target gain. VAT captures the system-level dynamics of value expression under alignment intervention, enabling evaluation of both intended improvements and unintended side effects. Using a controlled scenario-action dataset grounded in Schwartz value theory, we collect paired pre-post normative judgments and analyze alignment effects across models, values, and interventions. Results show that alignment often produces uneven and structured co-movement among values, revealing systematic trade-offs between target and non-target values. These effects are largely invisible under conventional target-only evaluation, but become evident via VAT, highlighting process-level alignment risks and offering new insights into the dynamic nature of value alignment in LLMs. Dataset and code are open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑