arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.03540cs.CV
收藏

SeTformer是视觉与语言领域所需之模型

SeTformer is What You Need for Vision and Language

East China Normal University(华东师范大学);Shanghai Jiaotong University(上海交通大学);ETS Montreal(蒙特利尔高等技术学院)

Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Michael Felsberg

AI总结:

提出SeTformer,用自最优传输(SeT)替换DPSA以提升计算效率与性能,在图像分类、检测、分割及语言任务中均达SOTA。

摘要:

点积自注意力(DPSA)是transformer的基本组件。然而,由于softmax操作带来的二次时间和内存复杂度,将其扩展到长序列(如文档或高分辨率图像)会变得计算成本极高。核方法被用于通过近似softmax来简化计算,但与softmax注意力相比,通常会导致性能下降。我们提出了SeTformer,一种新型transformer,其中DPSA被纯粹替换为自最优传输(SeT),以实现更好的性能和计算效率。SeT基于softmax的两个基本特性:维持非负注意力矩阵,以及使用非线性重加权机制来强调输入序列中的重要token。通过引入用于最优传输的核代价函数,SeTformer有效满足了这些特性。具体而言,在使用小型和基础尺寸模型时,SeTformer在ImageNet-1K上取得了84.7%和86.2%的惊人top-1准确率。在目标检测中,SeTformer-base以减少38%参数和29% FLOPs的优势,比FocalNet对应模型高出+2.2 mAP。在语义分割中,我们的基础尺寸模型以减少33%参数的优势超越NAT +3.5 mIoU。SeTformer还在GLUE基准的语言建模中取得了最先进的结果。这些发现突显了SeTformer在视觉和语言任务中的适用性。

英文摘要:

The dot product self-attention (DPSA) is a fundamental component of transformers. However, scaling them to long sequences, like documents or high-resolution images, becomes prohibitively expensive due to quadratic time and memory complexities arising from the softmax operation. Kernel methods are employed to simplify computations by approximating softmax but often lead to performance drops compared to softmax attention. We propose SeTformer, a novel transformer, where DPSA is purely replaced by Self-optimal Transport (SeT) for achieving better performance and computational efficiency. SeT is based on two essential softmax properties: maintaining a non-negative attention matrix and using a nonlinear reweighting mechanism to emphasize important tokens in input sequences. By introducing a kernel cost function for optimal transport, SeTformer effectively satisfies these properties. In particular, with small and basesized models, SeTformer achieves impressive top-1 accuracies of 84.7% and 86.2% on ImageNet-1K. In object detection, SeTformer-base outperforms the FocalNet counterpart by +2.2 mAP, using 38% fewer parameters and 29% fewer FLOPs. In semantic segmentation, our base-size model surpasses NAT by +3.5 mIoU with 33% fewer parameters. SeTformer also achieves state-of-the-art results in language modeling on the GLUE benchmark. These findings highlight SeTformer's applicability in vision and language tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑