arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.21187cs.CV
收藏

DSU-Net:基于DINOv2和SAM2并采用多尺度跨模型特征增强的改进U-Net模型

DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement

University of Chinese Academy of Sciences(中国科学院大学)

Yimin Xu, Fan Yang, Bin Xu

AI总结:

本文提出DSU-Net多尺度特征协作框架,通过DINOv2指导SAM2进行跨模型特征融合与轻量级适配器设计,在冻结基础模型参数下实现高效训练,并在伪装与显著目标检测等下游任务中超越现有SOTA方法。

摘要:

尽管大规模预训练基础模型(如Meta的Segment Anything Model (SAM)系列和DINOv2)在通用图像分割方面取得了显著进展,但它们在专门领域的性能仍受两个关键问题的限制:大模型参数导致的过高训练成本,以及对特定领域特征表示能力不足。本文提出了一种由DINOv2指导SAM2的多尺度特征协作框架,核心创新包含三方面:(1)在DINOv2和SAM2骨干网络之间建立特征协作机制,其中自监督模型提取的高维语义特征指导多尺度特征融合;(2)设计轻量级适配器模块和跨模态、跨层特征融合单元,在冻结基础模型参数的同时注入跨领域知识;(3)构建基于U-net的U型网络结构,利用注意力机制实现多粒度特征的自适应聚合解码。该框架在伪装目标检测和显著目标检测等下游任务中超越了现有最先进方法,且无需昂贵的训练过程。它为视觉图像分割的高效部署提供了技术途径,在图像分割的广泛下游任务和专门领域展现出显著应用价值。项目页面:https://github.com/CheneyXuYiMin/SAM2DINO-Seg

英文摘要:

Despite the significant advancements in general image segmentation achieved by large-scale pre-trained foundation models (such as Meta's Segment Any-thing Model (SAM) series and DINOv2), their performance in specialized fields remains limited by two critical issues: the excessive training costs due to large model parameters, and the insufficient ability to represent specific domain characteristics. This paper proposes a multi-scale feature collabora-tion framework guided by DINOv2 for SAM2, with core innovations in three aspects: (1) Establishing a feature collaboration mechanism between DINOv2 and SAM2 backbones, where high-dimensional semantic features extracted by the self-supervised model guide multi-scale feature fusion; (2) Designing lightweight adapter modules and cross-modal, cross-layer feature fusion units to inject cross-domain knowledge while freezing the base model parameters; (3) Constructing a U-shaped network structure based on U-net, which utilizes attention mechanisms to achieve adaptive aggregation decoding of multi-granularity features. This framework surpasses existing state-of-the-art meth-ods in downstream tasks such as camouflage target detection and salient ob-ject detection, without requiring costly training processes. It provides a tech-nical pathway for efficient deployment of visual image segmentation, demon-strating significant application value in a wide range of downstream tasks and specialized fields within image segmentation.Project page: https://github.com/CheneyXuYiMin/SAM2DINO-Seg

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑