arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.10258cs.LGcs.AIstat.ME
收藏

基于因果正则化的神经算法推理

Neural Algorithmic Reasoning with Causal Regularisation

Purdue University(普渡大学);DeepMind(深度思维)

Beatrice Bevilacqua, Kyriacos Nikiforou, Borja Ibarz, Ioana Bica, Michela Paganini, Charles Blundell, Jovana Mitrovic, Petar Veličković

AI总结:

本文提出 Hint-ReLIC 方法,利用算法中间计算不变性构建因果正则化与数据增强,在 CLRS 基准上显著提升神经算法推理的分布外泛化能力,OOD 测试性能最高提升 3 倍。

摘要:

近期关于神经算法推理的研究探索了神经网络的推理能力,有效证明了它们能够对来自训练分布的未见数据执行经典算法。然而,现有神经推理器在输入规模更大的分布外(OOD)测试数据上性能显著下降。本文提出了一个重要观察:对于许多不同的输入,算法会执行完全相同的中介计算。基于这一洞见,我们开发了数据增强方法,在给定算法中间轨迹的情况下,生成目标算法下一步轨迹完全相同的输入。我们通过一个由该观察推导出的自监督目标,并在因果图中形式化,确保此类输入在下一步预测中的不变性。我们证明了所提出的方法——称为 Hint-ReLIC——能够提升推理器的分布外泛化能力。我们在 CLRS 算法推理基准上评估了该方法,在 OOD 测试数据上展示了高达 3 倍的改进。

英文摘要:

Recent work on neural algorithmic reasoning has investigated the reasoning capabilities of neural networks, effectively demonstrating they can learn to execute classical algorithms on unseen data coming from the train distribution. However, the performance of existing neural reasoners significantly degrades on out-of-distribution (OOD) test data, where inputs have larger sizes. In this work, we make an important observation: there are many different inputs for which an algorithm will perform certain intermediate computations identically. This insight allows us to develop data augmentation procedures that, given an algorithm's intermediate trajectory, produce inputs for which the target algorithm would have exactly the same next trajectory step. We ensure invariance in the next-step prediction across such inputs, by employing a self-supervised objective derived by our observation, formalised in a causal graph. We prove that the resulting method, which we call Hint-ReLIC, improves the OOD generalisation capabilities of the reasoner. We evaluate our method on the CLRS algorithmic reasoning benchmark, where we show up to 3$\times$ improvements on the OOD test data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑