arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.07096cs.AI
收藏

在规划过程中拒绝幻觉状态目标

Rejecting Hallucinated State Targets during Planning

McGill University(麦吉尔大学);Mila (Quebec AI Institute)(蒙特利尔AI研究所);Google Deepmind(谷歌DeepMind)

Mingde Zhao, Tristan Sylvain, Romain Laroche, Doina Precup, Yoshua Bengio

AI总结:

针对计算决策智能体规划中生成模型产生不可行目标幻觉的问题,提出通过学习结合离策略规则、分布式架构与事后重标记数据增强的目标可行性评估器来识别并拒绝不可行目标,显著减少了智能体的妄想行为并提升了性能。

摘要:

在计算决策智能体的规划过程中,生成式或预测式模型常被用作“生成器”以提出代表预期或理想状态集合的“目标”。遗憾的是,学习到的模型不可避免地会产生不可行目标的幻觉,从而导致妄想行为与安全问题。我们首先调查了生成器可能产生幻觉的不可行目标类型。随后,我们设计了一种策略,通过学习目标可行性评估器来识别并拒绝不可行目标。为确保该评估器稳健且无妄想,我们采用了一项结合了离策略兼容学习规则、分布式架构以及基于事后重标记的数据增强的设计选择。将其附加到规划智能体上,所设计的评估器通过观察智能体与环境的交互及其生成器产生的目标进行学习,无需更改智能体或其生成器。我们的对照实验表明,针对各类现有的智能体,妄想行为显著减少且性能得到提升。

英文摘要:

In planning processes of computational decision-making agents, generative or predictive models are often used as "generators" to propose "targets" representing sets of expected or desirable states. Unfortunately, learned models inevitably hallucinate infeasible targets that can cause delusional behaviors and safety concerns. We first investigate the kinds of infeasible targets that generators can hallucinate. Then, we devise a strategy to identify and reject infeasible targets by learning a target feasibility evaluator. To ensure that the evaluator is robust and non-delusional, we adopted a design choice combining off-policy compatible learning rule, distributional architecture, and data augmentation based on hindsight relabeling. Attaching to a planning agent, the designed evaluator learns by observing the agent's interactions with the environment and the targets produced by its generator, without the need to change the agent or its generator. Our controlled experiments show significant reductions in delusional behaviors and performance improvements for various kinds of existing agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑