基于行列分解的Transformers双扁平化语义分割
Dual-Flattening Transformers through Decomposed Row and Column Queries for Semantic Segmentation
摘要：
对于诸如语义分割之类的密集预测任务，获得具有长程依赖性的高分辨率特征是至关重要的。为了从大小为$h\times w$（$hw\ll HW$）的低分辨率特征图生成大小为$H\times W$的高分辨率输出，朴素密集Transformer导致$\mathcal{O}(hwHW)$的难以处理的复杂性，限制了其在高分辨率密集预测上的应用。我们提出了一个双扁平Transformer（DFlatFormer），使高分辨率的输出，通过降低复杂性$\mathcal{O}(hw(H+W))$是几个数量级小于天真的密集Transformer。分解查询通过单独的Transformers可追踪地检索行和列注意力，并且它们的输出被组合以形成高分辨率的密集特征图。为此，从编码器馈送的输入序列被逐行和逐列扁平化，以通过分别保留其行和列结构来与分解的查询对齐。行和列Transformers也彼此交互以利用行和列之间的空间交叉来捕获它们的相互注意。我们还建议通过有效的分组和池化来执行注意力，以进一步降低模型的复杂性。在ADE 20 K和Cityscapes数据集上进行的大量实验证明了所提出的双扁平化Transformer架构具有更高mIoU的优越性。
英文摘要：
It is critical to obtain high resolution features with long range dependency for dense prediction tasks such as semantic segmentation. To generate high-resolution output of size $H\times W$ from a low-resolution feature map of size $h\times w$ ($hw\ll HW$), a naive dense transformer incurs an intractable complexity of $\mathcal{O}(hwHW)$, limiting its application on high-resolution dense prediction. We propose a Dual-Flattening Transformer (DFlatFormer) to enable high-resolution output by reducing complexity to $\mathcal{O}(hw(H+W))$ that is multiple orders of magnitude smaller than the naive dense transformer. Decomposed queries are presented to retrieve row and column attentions tractably through separate transformers, and their outputs are combined to form a dense feature map at high resolution. To this end, the input sequence fed from an encoder is row-wise and column-wise flattened to align with decomposed queries by preserving their row and column structures, respectively. Row and column transformers also interact with each other to capture their mutual attentions with the spatial crossings between rows and columns. We also propose to perform attentions through efficient grouping and pooling to further reduce the model complexity. Extensive experiments on ADE20K and Cityscapes datasets demonstrate the superiority of the proposed dual-flattening transformer architecture with higher mIoUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
