在注意力机制中注入晶格对称性先验以实现样本高效的抽象几何推理
Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoning
AI总结：
本文提出 LatFormer，通过卷积网络生成的软注意力掩码注入晶格对称性先验，在几何推理及 ARC/LARC 任务上显著提升样本效率。
摘要：
Abstraction and Reasoning Corpus（ARC，抽象与推理语料库）（Chollet，2019）及其最新的语言完备实例化版本 LARC，被视为迈向通用 AI 的重要一步。然而，即使是最先进的机器学习模型也难以在这些问题上取得有意义的性能，落后于非学习型方法。我们认为，解决这些任务需要极强的泛化能力，而这只有通过恰当利用核心知识先验才能实现。作为迈向该目标的一步，我们聚焦于几何先验，并提出 LatFormer，这是一种在注意力掩码中融入晶格对称性先验的模型。我们证明，对于超立方晶格的任何变换，都存在一个实现该群作用的二元注意力掩码。因此，我们的研究推动了对标准注意力机制的修改：使用由卷积网络生成的软掩码来缩放注意力权重。在合成几何推理上的实验表明，LatFormer 所需数据比标准注意力和 transformer 少 2 个数量级。此外，我们在包含几何先验的 ARC 和 LARC 任务上的结果提供了初步证据，表明这些复杂数据集并非超出深度学习模型的能力范围。
英文摘要：
The Abstraction and Reasoning Corpus (ARC) (Chollet, 2019) and its most recent language-complete instantiation (LARC) has been postulated as an important step towards general AI. Yet, even state-of-the-art machine learning models struggle to achieve meaningful performance on these problems, falling behind non-learning based approaches. We argue that solving these tasks requires extreme generalization that can only be achieved by proper accounting for core knowledge priors. As a step towards this goal, we focus on geometry priors and introduce LatFormer, a model that incorporates lattice symmetry priors in attention masks. We show that, for any transformation of the hypercubic lattice, there exists a binary attention mask that implements that group action. Hence, our study motivates a modification to the standard attention mechanism, where attention weights are scaled using soft masks generated by a convolutional network. Experiments on synthetic geometric reasoning show that LatFormer requires 2 orders of magnitude fewer data than standard attention and transformers. Moreover, our results on ARC and LARC tasks that incorporate geometric priors provide preliminary evidence that these complex datasets do not lie out of the reach of deep learning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
