发表机构
CETINIA, University Rey Juan Carlos（胡安·卡洛斯国王大学CETINIA中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究生成式人工智能中价值系统学习问题，提出将先前验证的方法用于此场景，基于成对偏好数据同时学习价值基础实现与价值系统表示，算法动态排序，评估显示其性能优、权衡小且可解释性强。
AI 中文摘要
价值感知人工智能系统需要对人类价值观进行明确的计算表示并将其聚合为价值系统，以便使其决策与人类决策保持一致。由于这种表示难以引出，价值学习试图通过观察人类行为来推断它们。本文解决了生成式人工智能中缺乏基于实际的价值学习方法的问题：现有方法通常在不了解价值对齐的多维结构的情况下复制人类偏好，或者缺乏有原则的价值系统引出方法。为了解决这些差距，我们将一种先前经过验证的价值系统学习方法应用于生成式人工智能设置，该方法基于成对的提示-响应偏好数据，同时学习：i）由多目标奖励模型给出的一组价值的基础实现，以及ii）以前述基础模型的加权线性标量化形式表示的价值系统。为确保学习到的价值系统基于连贯的价值表示，我们的算法动态地对基础学习过程进行优先级排序。我们在提示-响应偏好数据集上针对基线和当代方法评估了该方法。结果显示出具有竞争力的性能，与基线相比权衡最小，同时提高了可解释性。
英文摘要
Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are difficult to elicit, value learning seeks to infer them by observing human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human preferences without awareness of the multidimensional structure of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pairwise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and ii) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynamically prioritizes the grounding learning process. We evaluate the method against baselines and a contemporary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving explainability.
CommentsFull version of a to be published paper in proceedings of the 9th AAAI/ACM conference in AI, Ethics and Society (AIES 2026). Includes supplementary material. 20 pages, 2 figures
