arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.02584cs.LGstat.ML
收藏

数据点之间的自注意力:超越深度学习中的个体输入输出对

Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

University of Oxford(牛津大学);Cohere

Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, Yarin Gal

摘要:

我们挑战了大多数监督式深度学习的一个常见假设:模型仅根据其参数和单个输入的特征进行预测。为此,我们引入了一个通用的深度学习架构,它将整个数据集作为输入,而不是一次处理一个数据点。我们的方法使用自注意力来明确地推理数据点之间的关系,这可以被看作是使用参数注意力机制来实现非参数模型。然而,与传统的非参数模型不同,我们让模型从数据中端到端地学习如何利用其他数据点进行预测。从经验上讲,我们的模型解决了传统深度学习模型无法解决的跨数据点查找和复杂推理任务。我们在表格数据上显示了极具竞争力的结果,在CIFAR-10上显示了早期结果,并深入了解了模型如何利用点之间的相互作用。

英文摘要:

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑