arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.07547cs.CV
收藏

搜索TrioNet:结合卷积和局部及全局自注意

Searching for TrioNet: Combining Convolution with Local and Global Self-Attention

Zhejiang University(浙江大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Huaijin Pi, Huiyu Wang, Yingwei Li, Zizhang Li, Alan Yuille

摘要:

最近,自我注意力算子作为视觉模型的独立构建块表现出优越的性能。然而,现有的自注意力模型通常是手工设计的,从CNN修改而来,并且仅通过堆叠一个算子来获得。结合不同的自注意算子和卷积的更广泛的架构空间很少被探索。在本文中,我们探索这个新的架构空间与权重共享神经架构搜索(NAS)算法。结果架构被命名为TrioNet,用于组合卷积、局部自注意和全局(轴向)自注意运算符。为了在这个巨大的架构空间中有效地搜索,我们提出了分层采样来更好地训练超网。此外,我们提出了一种新的权重共享策略,多头共享,专门为多头自注意运营商。我们搜索到的TrioNet结合了自我注意力和卷积,在ImageNet分类上的表现优于所有独立模型,并且FLOP更少,其中自我注意力的表现优于卷积。此外,在各种小数据集上,我们观察到自我注意力模型的性能较差,但我们的TrioNet仍然能够匹配最佳算子,在这种情况下是卷积。我们的代码可在https://github.com/phj128/TrioNet上获得。

英文摘要:

Recently, self-attention operators have shown superior performance as a stand-alone building block for vision models. However, existing self-attention models are often hand-designed, modified from CNNs, and obtained by stacking one operator only. A wider range of architecture space which combines different self-attention operators and convolution is rarely explored. In this paper, we explore this novel architecture space with weight-sharing Neural Architecture Search (NAS) algorithms. The result architecture is named TrioNet for combining convolution, local self-attention, and global (axial) self-attention operators. In order to effectively search in this huge architecture space, we propose Hierarchical Sampling for better training of the supernet. In addition, we propose a novel weight-sharing strategy, Multi-head Sharing, specifically for multi-head self-attention operators. Our searched TrioNet that combines self-attention and convolution outperforms all stand-alone models with fewer FLOPs on ImageNet classification where self-attention performs better than convolution. Furthermore, on various small datasets, we observe inferior performance for self-attention models, but our TrioNet is still able to match the best operator, convolution in this case. Our code is available at https://github.com/phj128/TrioNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑