arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.06002cs.AIcs.LG
收藏

使用GNNs学习无监督的广义策略

Learning Generalized Policies Without Supervision Using GNNs

Linköping University(林雪平大学);Universitat Pompeu Fabra(庞培法布拉大学);Institució Catalana de Recerca i Estudis Avançats(加泰罗尼亚研究与高等教育学院)

Simon Ståhlberg, Blai Bonet, Hector Geffner

摘要:

我们认为,学习广义政策的经典规划域使用图神经网络的问题,从小的情况下,代表在解除的CNOPS。这个问题以前已经考虑过,但提出的神经架构是复杂的,结果往往是混合的。在这项工作中,我们使用一个简单而通用的GNN架构,旨在获得清晰的实验结果和更深入的理解:要么学习的价值函数中的策略贪婪在比训练中使用的实例更大的实例上实现接近100%的泛化,要么必须理解失败,并可能在逻辑上进行修复。为此,我们利用GNN的表达能力和一阶逻辑的$C_{2}$片段(即,具有2个变量和计数量词的FOL)之间建立的关系。例如,我们发现,具有需要更多表达功能的一般策略的域可以使用GNN解决,一旦状态被扩展为合适的“派生原子”编码角色组合和不适合$C_{2}$的传递闭包。这项工作遵循GNN方法,以监督的方式学习最优的一般策略(Stahlberg,Bonet,Geffner,2022);但学习的策略不再需要是最优的(这扩大了范围,因为许多规划领域没有一般的最优策略),并且在没有监督的情况下学习。有趣的是,旨在产生最优策略的基于值的强化学习方法并不总是产生泛化的策略,因为在最优规划是NP难的领域中,最优性和通用性的目标是冲突的。

英文摘要:

We consider the problem of learning generalized policies for classical planning domains using graph neural networks from small instances represented in lifted STRIPS. The problem has been considered before but the proposed neural architectures are complex and the results are often mixed. In this work, we use a simple and general GNN architecture and aim at obtaining crisp experimental results and a deeper understanding: either the policy greedy in the learned value function achieves close to 100% generalization over instances larger than those used in training, or the failure must be understood, and possibly fixed, logically. For this, we exploit the relation established between the expressive power of GNNs and the $C_{2}$ fragment of first-order logic (namely, FOL with 2 variables and counting quantifiers). We find for example that domains with general policies that require more expressive features can be solved with GNNs once the states are extended with suitable "derived atoms" encoding role compositions and transitive closures that do not fit into $C_{2}$. The work follows the GNN approach for learning optimal general policies in a supervised fashion (Stahlberg, Bonet, Geffner, 2022); but the learned policies are no longer required to be optimal (which expands the scope, as many planning domains do not have general optimal policies) and are learned without supervision. Interestingly, value-based reinforcement learning methods that aim to produce optimal policies, do not always yield policies that generalize, as the goals of optimality and generality are in conflict in domains where optimal planning is NP-hard.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑