发表机构
Lehigh University; Naval Research Laboratory; Interagens s.r.l.（理海大学; 海军研究实验室; Interagens有限责任公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文研究自主智能体在道德义务与用户任务冲突时的道德反叛行为，形式化五种智能体架构并在搜索救援领域验证，发现任务承诺保持是道德反叛的关键维度。
AI 中文摘要
自主智能体通常有义务遵循用户分配的任务。然而，严格服从可能与执行过程中出现的道德义务相冲突。本文研究道德反叛：自主智能体在道德上合理的情况下偏离用户分配任务的能力。我们形式化了五种智能体架构：一种不考虑道德义务而追求分配任务的非道德智能体，以及四种道德反叛智能体形式：（1）功利主义智能体，机会主义地最大化任务结果；（2）道义论智能体，强制执行规范性约束；（3）功利-道义（UD）智能体，将道义约束与功利推理相结合；（4）尽职智能体，额外保持对分配任务的承诺。我们在分层任务网络规划框架内实现这些架构，并在一个微型搜索与救援领域中评估它们，该领域揭示了分配任务完成、机会性救援和规范遵守之间的权衡。我们的实证结果表明，所提出的智能体在救援结果、分配任务完成和规范遵守之间表现出不同的权衡。特别是，任务承诺的保持成为道德反叛的一个重要维度，尽管UD和尽职智能体共享功利和道义基础，但它们的行为却显著不同。这些发现强调了在道德后果严重的环境中运行的自主智能体具有承诺感知的道德推理的重要性。
英文摘要
Autonomous agents are typically obliged to follow user-assigned tasks. However, strict obedience may conflict with moral obligations that arise during execution. This paper investigates \textbf{moral rebellion}: the ability of an autonomous agent to deviate from a user-assigned task when morally justified. We formalize five agent architectures: an \textit{amoral agent} that pursues assigned tasks without considering moral obligations, and four forms of \textbf{moral rebel agency}: (1) \textit{utilitarian agents} that opportunistically maximize task outcomes; (2) \textit{deontic} agents that enforce normative constraints; (3) \textit{utilitarian-deontic} (UD) agents that combine deontic constraints with utilitarian reasoning; and (4) \textit{dutiful} agents that additionally preserve commitments to assigned tasks. We implement these architectures within a hierarchical task network planning framework and evaluate them in a Mini Search-and-Rescue domain that exposes trade-offs among assigned-task completion, opportunistic rescue, and norm compliance. Our empirical results show that the proposed agents exhibit distinct trade-offs among rescue results, assigned-task completion, and norm compliance. In particular, the preservation of task commitments emerges as an important dimension of moral rebellion, for which the UD and dutiful agents produce substantially different behaviors despite their shared utilitarian and deontological foundations. These findings highlight the importance of commitment-aware moral reasoning for autonomous agents operating in morally consequential environments.
Comments17 pages, 1 table, 3 figures
Journal refHector Munoz-Avila, David W. Aha, and Paola Rizzo, "Moral Rebel Agents: Decision-Making Under Conflicting Obligations," Proceedings of the Thirteenth Annual Conference on Advances in Cognitive Systems, 2026
