MoralReason：基于推理级强化学习的大语言模型智能体可泛化道德决策对齐
MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
AI总结：
针对大语言模型智能体的分布外道德对齐难题，本文提出基于推理级强化学习的MoralReason方法，构建Moral-Reason-QA数据集并采用复合奖励的GRPO训练，实现了未见过场景下的道德决策对齐能力提升。
摘要：
大语言模型正日益影响人类的道德决策，但现有方法主要聚焦于评估其道德决策，而非主动引导。我们将该问题形式化为分布外道德对齐问题，即大语言模型智能体必须学会将一致的道德推理框架应用于训练分布之外的场景。我们推出了全新数据集Moral-Reason-QA，该数据集扩展了680个人工标注的高模糊度道德场景，附带覆盖功利主义、义务论、美德伦理学的特定框架推理轨迹，可在现实决策语境下对道德泛化能力开展系统性评估。我们的学习方法采用带复合奖励的组相对策略优化（Group Relative Policy Optimization），同时优化决策对齐度与特定框架推理过程，以助力智能体学习底层道德框架。实验结果表明，方法可成功泛化至未见过的道德场景，在分布外测试集上，功利主义框架的softmax归一化对齐分数提升0.757，义务论框架提升0.450。实验还揭示了训练挑战与具备潜力的未来研究方向。这些发现证实，可通过系统性训练让大语言模型智能体内化并将特定道德框架应用于新场景，为大语言模型深度融入人类决策流程背景下的AI安全奠定关键基础。
英文摘要：
Large language models are increasingly influencing human moral decisions, yet current approaches focus primarily on evaluating rather than actively steering their moral decisions. We formulate this as an out-of-distribution moral alignment problem, where LLM agents must learn to apply consistent moral reasoning frameworks to scenarios beyond their training distribution. We introduce Moral-Reason-QA, a novel dataset extending 680 human-annotated, high-ambiguity moral scenarios with framework-specific reasoning traces across utilitarian, deontological, and virtue ethics, enabling systematic evaluation of moral generalization in realistic decision contexts. Our learning approach employs Group Relative Policy Optimization with composite rewards that simultaneously optimize decision alignment and framework-specific reasoning processes to facilitate learning of the underlying moral frameworks. Experimental results demonstrate successful generalization to unseen moral scenarios, with softmax-normalized alignment scores improving by +0.757 for utilitarian and +0.450 for deontological frameworks when tested on out-of-distribution evaluation sets. The experiments also reveal training challenges and promising directions that inform future research. These findings establish that LLM agents can be systematically trained to internalize and apply specific moral frameworks to novel situations, providing a critical foundation for AI safety as language models become more integrated into human decision-making processes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
