近期提出的Segment Anything Model(SAM,分割一切模型)是自然图像分割领域的一项重大进展,展现出强大的零样本性能,适用于各类下游图像分割任务。然而,由于自然图像与红外图像之间存在显著的域差距,直接将预训练SAM用于红外小目标检测(Infrared Small Target Detection, IRSTD)任务无法取得令人满意的性能。与可见光相机不同,热成像仪通过捕获红外辐射来呈现物体的温度分布,小目标通常在物体边界处表现出细微的温度过渡。为解决这一问题,我们提出了用于IRSTD的IRSAM模型,该模型改进了SAM的编码器-解码器架构,以学习红外小目标更优的特征表示。具体而言,我们设计了一个基于Perona-Malik扩散(PMD)的模块,并将其融入SAM编码器的多个层级,以帮助编码器在抑制噪声的同时捕获关键的结构特征。此外,我们设计了粒度感知解码器(Granularity-Aware Decoder, GAD),用于融合来自编码器的多粒度特征,以捕获长距离建模中可能丢失的结构信息。在NUAA-SIRST、NUDT-SIRST和IRSTD-1K等公开数据集上开展的大量实验,验证了IRSAM的设计合理性,以及其相较于代表性的当前最优方法的显著优越性。源代码已公开于:github.com/IPIC-Lab/IRSAM。
英文摘要:
The recent Segment Anything Model (SAM) is a significant advancement in natural image segmentation, exhibiting potent zero-shot performance suitable for various downstream image segmentation tasks. However, directly utilizing the pretrained SAM for Infrared Small Target Detection (IRSTD) task falls short in achieving satisfying performance due to a notable domain gap between natural and infrared images. Unlike a visible light camera, a thermal imager reveals an object's temperature distribution by capturing infrared radiation. Small targets often show a subtle temperature transition at the object's boundaries. To address this issue, we propose the IRSAM model for IRSTD, which improves SAM's encoder-decoder architecture to learn better feature representation of infrared small objects. Specifically, we design a Perona-Malik diffusion (PMD)-based block and incorporate it into multiple levels of SAM's encoder to help it capture essential structural features while suppressing noise. Additionally, we devise a Granularity-Aware Decoder (GAD) to fuse the multi-granularity feature from the encoder to capture structural information that may be lost in long-distance modeling. Extensive experiments on the public datasets, including NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, validate the design choice of IRSAM and its significant superiority over representative state-of-the-art methods. The source code are available at: github.com/IPIC-Lab/IRSAM.
Shanghai Jiao Tong University(上海交通大学);WeChat, Tencent Inc(腾讯公司微信事业部);University of Science and Technology of China(中国科学技术大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学)
SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);College of Computer Science, East China Normal University(华东师范大学计算机科学学院);Department of Data Science and AI, Faculty of IT, Monash University(莫纳什大学信息技术学院数据科学与人工智能系);College of Computer Science, Sichuan University(四川大学计算机学院)
Advancing Weakly-Supervised Change Detection in Satellite Images via Adversarial Class Prompting
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University(信息工程测绘遥感国家重点实验室,武汉大学);School of Computer Science and Technology, Anhui University(计算机科学与技术学院,安徽大学);Graduate School of Frontier Sciences, University of Tokyo(前沿科学研究院,东京大学);Institute of Geodesy and Photogrammetry, ETH Zürich(测绘学研究院,苏黎世联邦理工学院);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学)
Advancing Marine Research: UWSAM Framework and UIIS10K Dataset for Precise Underwater Instance Segmentation
School of Computer Science and Technology, Hainan University(海南大学计算机科学与技术学院);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);College of Computer Science, Nankai University(南开大学计算机学院);School of Computer Science and Artificial Intelligence, Zhengzhou University(郑州大学计算机科学与人工智能学院);Department of Automation, Shanghai Jiao Tong University(上海交通大学自动化学院);School of Data Science, Lingnan University(岭南大学数据科学学院)
针对 SAM 在水下实例分割中的领域适配与计算开销问题,本文构建 UIIS10K 数据集,并提出结合 MG-UKD 与 EUPG 的高效 UWSAM,在多个数据集上显著优于现有方法。
Encoding Structural Constraints into Segment Anything Models via Probabilistic Graphical Models
The George Washington University(乔治·华盛顿大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究所,中国科学院);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校)
Going Beyond U-Net: Assessing Vision Transformers for Semantic Segmentation in Microscopy Image Analysis
Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究所);Center for Artificial Intelligence, Technical University of Applied Sciences Würzburg-Schweinfurt(维尔茨堡-施韦因富特应用技术大学人工智能中心);STACC
E-BayesSAM: Efficient Bayesian Adaptation of SAM with Self-Optimizing KAN-Based Interpretation for Uncertainty-Aware Ultrasonic Segmentation
the National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound(医学超声关键技术工程实验室);Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging(广东省生物医学测量与超声成像重点实验室);School of Biomedical Engineering(生物医学工程学院);Shenzhen University Medical School(深圳大学医学院);Shenzhen University(深圳大学);the Institute of Maternal and Child Medicine(妇幼医学研究院);Shenzhen Maternity and Child Healthcare Hospital(深圳妇幼保健院);Southern Medical University(南方医科大学);Medical AI Lab(医学人工智能实验室);the Department of Biomedical Engineering(生物医学工程系);the Department of Computer and Data Science(计算机与数据科学系);Case Western Reserve University(凯斯西储大学)
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
Thomas Jefferson High School for Science and Technology(托马斯·杰弗逊高中科学与技术学校);Mass Eye and Ear, Harvard Medical School(麻省眼耳研究所,哈佛医学院);Harvard Ophthalmology AI Lab, Schepens Eye Research Institute of Mass Eye and Ear, Harvard Medical School(哈佛眼科学人工智能实验室,麻省眼耳研究所的谢普恩斯眼研究学院,哈佛医学院)
Towards SAMBA: Segment Anything Model for Brain Tumor Segmentation in Sub-Sharan African Populations
Nile University(尼罗大学);Friedrich-Alexander-Universität(埃尔朗根-纽伦堡大学);Makerere University(马凯雷雷大学);Malawi University of Science and Technology(马拉维科技大学);Medical Artificial Intelligence Laboratory (MAI Lab)(医学人工智能实验室(MAI Lab));Lawson Health Research Institute(劳森健康研究院);University of British Columbia(不列颠哥伦比亚大学);Montreal Neurological Institute, McGill University(麦吉尔大学蒙特利尔神经学研究所);University of Cape Town(开普敦大学)
The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification
Conservation X Labs (CXL)(Conservation X Labs);Meta;University of Bristol(布里斯托大学);Hasso Plattner Institute(哈索罗滕堡研究所);University of Oviedo(奥维多大学);Osa Conservation(Osa保护);Senckenberg Museum of Natural History(Senckenberg自然史博物馆);Max Planck Institute for Evolutionary Anthropology(马克斯·普朗克进化人类学研究所);Climate Corridors(气候走廊)
SAM-Swin: SAM-Driven Dual-Swin Transformers with Adaptive Lesion Enhancement for Laryngo-Pharyngeal Tumor Detection
College of Big Data and Internet, Shenzhen Technology University(深圳技术大学大数据与互联网学院);The First Affiliated Hospital, Sun Yat-Sen University(中山大学附属第一医院);Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University(华南师范大学阿伯丁数据科学与人工智能学院)