arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.09419cs.LG
收藏

Compositional Attention:解开搜索和检索的纠缠

Compositional Attention: Disentangling Search and Retrieval

Mila(米拉);Université de Montréal(蒙特利尔大学)

Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, Guillaume Lajoie

摘要:

多头、键值关注是广泛成功的Transformer模型及其变体的支柱。这种注意力机制使用多个并行的键值注意力块(称为头),每个块执行两个基本计算:(1)搜索-经由查询-键交互从集合中选择相关实体,以及(2)检索-经由值矩阵从所选实体中提取相关特征。重要的是,标准的注意力头在搜索和检索之间有一个严格的映射。在这项工作中,我们首先强调了这种静态配对的性质如何可能:(a)导致在某些任务中学习冗余参数,以及(b)阻碍泛化。为了缓解这个问题,我们提出了一种新的注意力机制,称为组合注意力,它取代了标准的头部结构。所提出的机制解开搜索和检索,并组成他们在一个动态的,灵活的和上下文相关的方式,通过一个额外的软竞争阶段之间的查询键组合和值配对。通过一系列的数值实验,我们表明,它优于标准的多头注意力的各种任务,包括一些分布设置。通过我们的定性分析,我们表明,成分注意导致动态专业化的基础上所需的检索类型。我们提出的机制概括了多头注意力,允许搜索和检索的独立缩放,并且可以很容易地实现在任何网络架构中的标准注意力头的替代。

英文摘要:

Multi-head, key-value attention is the backbone of the widely successful Transformer model and its variants. This attention mechanism uses multiple parallel key-value attention blocks (called heads), each performing two fundamental computations: (1) search - selection of a relevant entity from a set via query-key interactions, and (2) retrieval - extraction of relevant features from the selected entity via a value matrix. Importantly, standard attention heads learn a rigid mapping between search and retrieval. In this work, we first highlight how this static nature of the pairing can potentially: (a) lead to learning of redundant parameters in certain tasks, and (b) hinder generalization. To alleviate this problem, we propose a novel attention mechanism, called Compositional Attention, that replaces the standard head structure. The proposed mechanism disentangles search and retrieval and composes them in a dynamic, flexible and context-dependent manner through an additional soft competition stage between the query-key combination and value pairing. Through a series of numerical experiments, we show that it outperforms standard multi-head attention on a variety of tasks, including some out-of-distribution settings. Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed. Our proposed mechanism generalizes multi-head attention, allows independent scaling of search and retrieval, and can easily be implemented in lieu of standard attention heads in any network architecture.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑