什么是人类价值观，以及我们如何让AI对齐人类价值观？
What are human values, and how do we align AI to them?
AI总结：
该研究将AI对齐人类价值观问题拆分为三部分，聚焦价值观提取与整合，提出受价值哲学启发的MGE方法，经500名美国人测试，在6项标准上均提升了对齐效果，还能自然凸显相关群体的专家价值观。
摘要：
目前学界逐渐形成共识：我们需要让AI系统对齐人类价值观（Gabriel, 2020; Ji等, 2024），但目前尚不清楚如何在实践中将其应用于语言模型。我们将“对齐人类价值观”的问题拆分为三个部分：第一，从人群中引导提取价值观；第二，将这些价值观协调整合为机器学习模型训练的对齐目标；第三，实际训练模型。在本文中，我们聚焦于前两个部分，并提出问题：有哪些“好”的方法可以将关于价值观的多样化人类输入整合为语言模型对齐的目标？为回答这一问题，我们首先定义了6项标准，我们认为一个对齐目标要让模型行为符合人类价值观，必须满足这些标准。随后我们提出了一种用于引导提取和协调价值观的流程，名为Moral Graph Elicitation（MGE，道德图谱引导法），该方法利用大语言模型就特定情境下的价值观对参与者进行访谈；我们的方法受到Taylor（1977）、Chang（2004）等人提出的价值哲学的启发。我们在500名具有代表性的美国样本中测试了MGE，测试采用了3个刻意选取的有争议性提示（例如关于堕胎的建议）。结果表明，MGE在全部6项标准上都有潜力提升模型对齐效果。例如，几乎所有参与者（89.1%）认为该流程很好地代表了自己的观点，且89%的参与者认为最终的道德图谱是公平的，即便他们自身的价值观没有被投票为最明智的选项。我们的流程通常会让“专家”价值观（例如曾寻求过堕胎建议的女性的价值观）升至道德图谱的顶端，且无需预先定义谁属于专家。
英文摘要：
There is an emerging consensus that we need to align AI systems with human values (Gabriel, 2020; Ji et al., 2024), but it remains unclear how to apply this to language models in practice. We split the problem of "aligning to human values" into three parts: first, eliciting values from people; second, reconciling those values into an alignment target for training ML models; and third, actually training the model. In this paper, we focus on the first two parts, and ask the question: what are "good" ways to synthesize diverse human inputs about values into a target for aligning language models? To answer this question, we first define a set of 6 criteria that we believe must be satisfied for an alignment target to shape model behavior in accordance with human values. We then propose a process for eliciting and reconciling values called Moral Graph Elicitation (MGE), which uses a large language model to interview participants about their values in particular contexts; our approach is inspired by the philosophy of values advanced by Taylor (1977), Chang (2004), and others. We trial MGE with a representative sample of 500 Americans, on 3 intentionally divisive prompts (e.g. advice about abortion). Our results demonstrate that MGE is promising for improving model alignment across all 6 criteria. For example, almost all participants (89.1%) felt well represented by the process, and (89%) thought the final moral graph was fair, even if their value wasn't voted as the wisest. Our process often results in "expert" values (e.g. values from women who have solicited abortion advice) rising to the top of the moral graph, without defining who is considered an expert in advance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
