arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04838cs.AIcs.CYcs.LG
收藏

考虑可接受性松弛的学习价值对齐策略的算法

Algorithms for learning value-aligned policies considering admissibility relaxation

CETINIA;Universidad Rey Juan Carlos(胡安·卡洛斯国王大学)

Andrés Holgado-Sánchez, Joaquín Arias, Holger Billhardt, Sascha Ossowski

AI总结:

针对价值对齐策略中可接受性约束松弛带来的计算复杂度问题,本文提出基于约束强化学习的ε-ADQL和ε-CADQL两种算法,并在干旱情景水资源分配问题中验证了其效率。

摘要:

新兴的“价值意识工程”领域主张软件智能体和系统应当具备价值意识,即它们必须依据人类价值观做出决策。在此背景下,这类智能体必须能够显式地推理不同行动方案与这些价值观的对齐程度。为此,价值观通常被建模为对状态或动作的偏好,随后对这些偏好进行聚合,以确定与某一价值最大程度对齐的动作序列。最近,该层面上的额外价值可接受性约束也已被纳入考虑。然而,往往需要这些约束的松弛版本,这显著增加了计算价值对齐策略的复杂度。为了获得在考虑可接受性松弛的情况下做出价值对齐决策的高效算法,我们提出使用学习技术,特别是采用了约束强化学习算法。在本文中,我们提出了两种算法:用于基于局部对齐策略的ε-ADQL,以及其面向决策序列的扩展ε-CADQL。我们在干旱情景下的水资源分配问题中验证了它们的效率。

英文摘要:

The emerging field of \emph{value awareness engineering} claims that software agents and systems should be value-aware, i.e. they must make decisions in accordance with human values. In this context, such agents must be capable of explicitly reasoning as to how far different courses of action are aligned with these values. For this purpose, values are often modelled as preferences over states or actions, which are then aggregated to determine the sequences of actions that are maximally aligned with a certain value. Recently, additional value admissibility constraints at this level have been considered as well. However, often relaxed versions of these constraints are needed, and this increases considerably the complexity of computing value-aligned policies. To obtain efficient algorithms that make value-aligned decisions considering admissibility relaxation, we propose the use of learning techniques, in particular, we have used constrained reinforcement learning algorithms. In this paper, we present two algorithms, $ε\text{-}ADQL$ for strategies based on local alignment and its extension $ε\text{-}CADQL$ for a sequence of decisions. We have validated their efficiency in a water distribution problem in a drought scenario.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑