探索用于可控多元对齐的思维链推理
Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment
AI总结：
针对大语言模型价值观统一、难以适配多元观点需求的问题，探究思维链推理技术构建可控多元对齐模型的效果，对比多种方法后发现RLVR性能最优且样本效率高，并分析了思维链轨迹的忠实度与安全性。
摘要：
大语言模型（LLM）通常被训练为反映一套相对统一的价值观，这限制了它们在需要理解人类细微差别的观点的任务中的适用性。近期研究强调了让大语言模型支持可控多元性的重要性——即采取特定观点并使生成的输出与之对齐的能力。在本研究中，我们探究思维链（Chain-of-Thought, CoT）推理技术是否可用于构建可控多元模型。我们探索了多种方法，包括CoT提示（CoT prompting）、基于人工撰写的CoT进行微调、基于合成解释进行微调，以及带可验证奖励的强化学习（Reinforcement Learning with Verifiable Rewards, RLVR）。我们使用Value Kaleidoscope和OpinionQA数据集评估这些方法。在研究的所有方法中，RLVR始终优于其他方法，并且展现出出色的训练样本效率。我们进一步从忠实度和安全性角度分析了生成的CoT轨迹。
英文摘要：
Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. Recent research has underscored the importance of enabling LLMs to support steerable pluralism -- the capacity to adopt a specific perspective and align generated outputs with it. In this work, we investigate whether Chain-of-Thought (CoT) reasoning techniques can be applied to building steerable pluralistic models. We explore several methods, including CoT prompting, fine-tuning on human-authored CoT, fine-tuning on synthetic explanations, and Reinforcement Learning with Verifiable Rewards (RLVR). We evaluate these approaches using the Value Kaleidoscope and OpinionQA datasets. Among the methods studied, RLVR consistently outperforms others and demonstrates strong training sample efficiency. We further analyze the generated CoT traces with respect to faithfulness and safety.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
