arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19613cs.CV
收藏

用于少样本分割的混合Mamba

Hybrid Mamba for Few-Shot Segmentation

S-Lab, Nanyang Technological University(新加坡南洋理工大学S-Lab);Peking University(北京大学);Singapore University of Technology and Design(新加坡科技设计大学);University of Cologne(科隆大学);SenseTime Research(商汤科技研究院)

Qianxiong Xu, Xuanyi Liu, Lanyun Zhu, Guosheng Lin, Cheng Long, Ziyue Li, Rui Zhao

AI总结:

本文提出混合Mamba网络HMNet,通过支持重述和查询拦截两个模块解决少样本分割中支持遗忘与类内差距问题,有效利用支持信息并在两个公开基准上取得优越性能。

摘要:

许多少样本分割(FSS)方法使用交叉注意力将支持前景(FG)融合到查询特征中,而不考虑其二次复杂度。最近的一项进展Mamba也能很好地捕获序列内依赖关系,但复杂度仅为线性。因此,我们旨在设计一种交叉(类注意力)Mamba来捕获FSS中的序列间依赖关系。一个简单的想法是对支持特征进行扫描,将其选择性地压缩到隐藏状态中,然后将该隐藏状态作为初始隐藏状态,依次扫描查询特征。然而,这种方法存在以下问题:(1)支持遗忘问题:查询特征在扫描时也会被逐渐压缩,因此隐藏状态中的支持特征不断减少,许多查询像素无法融合足够的支持特征;(2)类内差距问题:查询前景本质上与自身更相似,而不是与支持前景更相似,即查询可能更倾向于不融合支持特征,而是融合隐藏状态中自身的特征,然而FSS的成功依赖于对支持信息的有效利用。为了解决这些问题,我们设计了一个混合Mamba网络(HMNet),包括:(1)支持重述Mamba,在扫描查询时定期重述支持特征,使隐藏状态始终包含丰富的支持信息;(2)查询拦截Mamba,禁止查询像素之间的相互交互,并鼓励它们从隐藏状态中融合更多支持特征。因此,支持信息得到更好的利用,从而带来更好的性能。在两个公开基准上进行了大量实验,证明了HMNet的优越性。代码可在https://github.com/Sam1224/HMNet获取。

英文摘要:

Many few-shot segmentation (FSS) methods use cross attention to fuse support foreground (FG) into query features, regardless of the quadratic complexity. A recent advance Mamba can also well capture intra-sequence dependencies, yet the complexity is only linear. Hence, we aim to devise a cross (attention-like) Mamba to capture inter-sequence dependencies for FSS. A simple idea is to scan on support features to selectively compress them into the hidden state, which is then used as the initial hidden state to sequentially scan query features. Nevertheless, it suffers from (1) support forgetting issue: query features will also gradually be compressed when scanning on them, so the support features in hidden state keep reducing, and many query pixels cannot fuse sufficient support features; (2) intra-class gap issue: query FG is essentially more similar to itself rather than to support FG, i.e., query may prefer not to fuse support features but their own ones from the hidden state, yet the success of FSS relies on the effective use of support information. To tackle them, we design a hybrid Mamba network (HMNet), including (1) a support recapped Mamba to periodically recap the support features when scanning query, so the hidden state can always contain rich support information; (2) a query intercepted Mamba to forbid the mutual interactions among query pixels, and encourage them to fuse more support features from the hidden state. Consequently, the support information is better utilized, leading to better performance. Extensive experiments have been conducted on two public benchmarks, showing the superiority of HMNet. The code is available at https://github.com/Sam1224/HMNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑