OmniSAM：用于全景语义分割无监督域适应的全向分割一切模型
OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation
AI总结：
针对全景图像视场差异和缺乏像素级语义理解的问题，提出OmniSAM框架，首次将SAM2应用于全景语义分割，通过图像块序列和记忆机制提取跨视场依赖，并引入原型适应模块，大幅超越现有方法。
摘要：
Segment Anything Model 2 (SAM2) 已成为各种针孔成像分割任务中的强大基础模型。然而，将其应用于360度领域时，针孔图像（70度×70度）与全景图像（180度×360度）之间显著的视场（FoV）差异带来了独特的挑战。该应用的两个主要问题包括：1）域间大视场差异带来的不可避免失真和物体变形；2）原始SAM2无法提供像素级语义理解。为解决这些问题，我们提出了新颖的OmniSAM框架，首次尝试将SAM2应用于全景语义分割。具体而言，为弥合第一个差距，OmniSAM首先将全景图划分为图像块序列。这些图像块随后被视作视频分割任务中的图像序列处理。我们利用SAM2的记忆机制提取跨图像块对应关系，嵌入跨视场依赖性，提升特征连续性和掩膜边界预测一致性。针对第二个差距，OmniSAM微调预训练图像编码器并重新利用掩膜解码器进行语义预测。同时引入基于视场的原型适应模块及动态伪标签更新机制，促进记忆与主干特征对齐，从而提升不同规模源模型的泛化能力。大量实验表明，OmniSAM大幅超越现有最先进方法，如在SPin8-to-SPan8上达79.06%（+10.22%），在CS13-to-DP13上达62.46%（+6.58%）。
英文摘要：
Segment Anything Model 2 (SAM2) has emerged as a strong base model in various pinhole imaging segmentation tasks. However, when applying it to $360^\circ$ domain, the significant field-of-view (FoV) gap between pinhole ($70^\circ \times 70^\circ$) and panoramic images ($180^\circ \times 360^\circ$) poses unique challenges. Two major concerns for this application includes 1) inevitable distortion and object deformation brought by the large FoV disparity between domains; 2) the lack of pixel-level semantic understanding that the original SAM2 cannot provide. To address these issues, we propose a novel OmniSAM framework, which makes the first attempt to apply SAM2 for panoramic semantic segmentation. Specifically, to bridge the first gap, OmniSAM first divides the panorama into sequences of patches. These patches are then treated as image sequences in similar manners as in video segmentation tasks. We then leverage the SAM2's memory mechanism to extract cross-patch correspondences that embeds the cross-FoV dependencies, improving feature continuity and the prediction consistency along mask boundaries. For the second gap, OmniSAM fine-tunes the pretrained image encoder and reutilize the mask decoder for semantic prediction. An FoV-based prototypical adaptation module with dynamic pseudo label update mechanism is also introduced to facilitate the alignment of memory and backbone features, thereby improving model generalization ability across different sizes of source models. Extensive experimental results demonstrate that OmniSAM outperforms the state-of-the-art methods by large margins, e.g., 79.06% (+10.22%) on SPin8-to-SPan8, 62.46% (+6.58%) on CS13-to-DP13.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
