arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.05613cs.CV
收藏

窗口注意力存在缺陷:如何避免插值位置嵌入

Window Attention is Bugged: How not to Interpolate Position Embeddings

Georgia Tech(佐治亚理工学院)

Daniel Bolya, Chaitanya Ryali, Judy Hoffman, Christoph Feichtenhofer

AI总结:

本研究发现窗口注意力与插值位置嵌入组合存在性能缺陷,提出绝对窗口位置嵌入策略修复Hiera和ViTDet的问题,结合得到的HieraDet在仅ImageNet-1k预训练的模型中达COCO 61.7 box mAP的SOTA水平。

摘要:

窗口注意力、位置嵌入和高分辨率微调是计算机视觉现代Transformer时代的核心概念。然而我们发现,直接组合这些几乎通用的组件会对性能产生不利影响。问题很简单:在使用窗口注意力时对位置嵌入进行插值是错误的。我们研究了两种具备这三个组件的SOTA方法——Hiera和ViTDet,发现二者确实都受此缺陷影响。为解决该问题,我们提出一种简单的绝对窗口位置嵌入策略,可彻底修复Hiera中的缺陷,还能同时提升ViTDet模型的速度与性能。最后我们将二者结合得到HieraDet,其在COCO上达到61.7的box mAP,成为仅使用ImageNet-1k预训练的模型中的SOTA。这一切都源于本质上仅3行代码的缺陷修复,我们将其命名为“absolute win”。

英文摘要:

Window attention, position embeddings, and high resolution finetuning are core concepts in the modern transformer era of computer vision. However, we find that naively combining these near ubiquitous components can have a detrimental effect on performance. The issue is simple: interpolating position embeddings while using window attention is wrong. We study two state-of-the-art methods that have these three components, namely Hiera and ViTDet, and find that both do indeed suffer from this bug. To fix it, we introduce a simple absolute window position embedding strategy, which solves the bug outright in Hiera and allows us to increase both speed and performance of the model in ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k pretraining. This all stems from what is essentially a 3 line bug fix, which we name "absolute win".

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑