Survey-to-Behavior：通过调查问卷实现大语言模型中人类价值观的下游对齐
Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questions
AI总结：
本文提出通过微调模型回答价值观调查问卷来实现人类价值观的下游对齐，实验表明该方法既能改变域内问卷回答，也能在情境化道德判断和文本冒险游戏等域外任务中产生实质性行为转变。
摘要：
大语言模型隐式地编码了对人类价值观的偏好，然而引导这些偏好通常需要大规模训练数据。在本工作中，我们研究一种简单的方法：能否通过训练模型按照要求回答价值观调查问卷，来可靠地修改其在下游行为中的价值体系？我们首先通过让若干开源大语言模型对涵盖20种不同人类价值观的一系列价值相关描述进行评分，构建它们的价值观画像，并将其作为后续实验的基线。随后，我们研究模型的价值体系是否可以通过在价值观调查问卷上进行微调来加以控制。我们从两个方面评估微调对模型行为的影响：第一，评估模型在域内、留存的调查问卷问题上答案的变化；第二，评估模型在域外场景（情境性场景）中的行为是否发生变化。为此，我们基于Reddit帖子构建了一个情境化道德判断数据集，并评估模型在基于文本的冒险游戏中的行为变化。我们证明，这种简单的方法不仅能改变模型对域内调查问卷问题的回答，还能在隐式的下游任务行为中产生实质性转变（价值对齐）。
英文摘要：
Large language models implicitly encode preferences over human values, yet steering them often requires large training data. In this work, we investigate a simple approach: Can we reliably modify a model's value system in downstream behavior by training it to answer value survey questions accordingly? We first construct value profiles of several open-source LLMs by asking them to rate a series of value-related descriptions spanning 20 distinct human values, which we use as a baseline for subsequent experiments. We then investigate whether the value system of a model can be governed by fine-tuning on the value surveys. We evaluate the effect of finetuning on the model's behavior in two ways; first, we assess how answers change on in-domain, held-out survey questions. Second, we evaluate whether the model's behavior changes in out-of-domain settings (situational scenarios). To this end, we construct a contextualized moral judgment dataset based on Reddit posts and evaluate changes in the model's behavior in text-based adventure games. We demonstrate that our simple approach can not only change the model's answers to in-domain survey questions, but also produces substantial shifts (value alignment) in implicit downstream task behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
