arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15826cs.LGcs.AI
收藏

面向可辩护序贯决策的奖励设计

Reward Design for Justifiable Sequential Decision-Making

Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Saarland University(萨尔大学)

Aleksa Sukovic, Goran Radanovic

AI总结:

本文提出用零和辩论博弈构建奖励模型以量化决策可辩护性,并训练可辩护策略,在脓毒症治疗决策中证明该反馈能显著提升策略可辩护性且几乎不损失性能,同时与理想裁判反馈相当。

摘要:

赋予智能体利用支持性证据为其所做决策进行辩护的能力,是负责任决策的基石之一。此外,确保辩护理由符合人类期望和社会规范至关重要,尤其是在医疗保健等高风险场景中。在本工作中,我们提出为强化学习智能体使用一种基于辩论的奖励模型,其中零和辩论博弈的结果量化了某个决策在特定状态下的可辩护性。该奖励模型随后被用于训练可辩护策略,使策略做出的决策能够更容易地获得支持性证据的佐证。在辩论博弈中,两个论证型智能体轮流为两个相互竞争的决策提供支持性证据。给定所提出的证据后,一个人类裁判的代理模型评估哪个决策得到了更好的辩护。我们展示了该方法在为脓毒症患者开具并辩护治疗决策的策略学习中的潜力。我们表明,与仅从环境奖励中获得的策略相比,用基于辩论的奖励模型生成的反馈信号增强奖励后,所得到的策略受到裁判的高度青睐,同时几乎没有牺牲任何性能。此外,就训练策略的整体性能和可辩护性而言,基于辩论的反馈与从理想裁判代理获得的反馈相当,后者利用状态中编码的完整信息来评估决策。这表明辩论博弈输出了状态中所包含的、与评估决策最相关的关键信息,从而证实了将我们的方法与人在回路评估相结合的实际可行性。最后,我们展示了通过多智能体辩论训练的智能体学会了提出能够抵御反驳且与人类偏好高度一致的证据。

英文摘要:

Equipping agents with the capacity to justify made decisions using supporting evidence represents a cornerstone of accountable decision-making. Furthermore, ensuring that justifications are in line with human expectations and societal norms is vital, especially in high-stakes situations such as healthcare. In this work, we propose the use of a debate-based reward model for reinforcement learning agents, where the outcome of a zero-sum debate game quantifies the justifiability of a decision in a particular state. This reward model is then used to train a justifiable policy, whose decisions can be more easily corroborated with supporting evidence. In the debate game, two argumentative agents take turns providing supporting evidence for two competing decisions. Given the proposed evidence, a proxy of a human judge evaluates which decision is better justified. We demonstrate the potential of our approach in learning policies for prescribing and justifying treatment decisions of septic patients. We show that augmenting the reward with the feedback signal generated by the debate-based reward model yields policies highly favored by the judge when compared to the policy obtained solely from the environment rewards, while hardly sacrificing any performance. Moreover, in terms of the overall performance and justifiability of trained policies, the debate-based feedback is comparable to the feedback obtained from an ideal judge proxy that evaluates decisions using the full information encoded in the state. This suggests that the debate game outputs key information contained in states that is most relevant for evaluating decisions, which in turn substantiates the practicality of combining our approach with human-in-the-loop evaluations. Lastly, we showcase that agents trained via multi-agent debate learn to propose evidence that is resilient to refutations and closely aligns with human preferences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑