分割基础模型的前所未有的发展已成为计算机视觉领域的主导力量，在广泛的自然图像和视频中引入了大量此前未被探索的能力。具体而言，Segment Anything Model (SAM) 标志着提示驱动范式向图像分割领域的显著扩展。最近推出的 SAM2 有效地将原始 SAM 扩展为流式处理方式，并在视频分割中展现出强大的性能。然而，由于自然图像与医学图像之间存在显著差异，这些模型在生物医学图像和视频上的有效性仍在探索中。本文概述了近期在将 SAM2 应用于和适配生物医学图像和视频方面的努力。研究结果表明，尽管 SAM2 在减少标注负担和实现零样本分割方面展现出前景，但其性能在不同的数据集和任务中存在差异。通过适配和微调来解决自然图像与医学图像之间的领域差距，对于在临床应用中充分释放 SAM2 的潜力至关重要。为了支持持续的研究工作，我们维护着一个活跃的存储库，其中包含最新的 SAM 和 SAM2 相关论文及项目，网址为 https://github.com/YichiZhang98/SAM4MIS。
英文摘要：
The unprecedented developments in segmentation foundational models have become a dominant force in the field of computer vision, introducing a multitude of previously unexplored capabilities in a wide range of natural images and videos. Specifically, the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation. The recent introduction of SAM2 effectively extends the original SAM to a streaming fashion and demonstrates strong performance in video segmentation. However, due to the substantial distinctions between natural and medical images, the effectiveness of these models on biomedical images and videos is still under exploration. This paper presents an overview of recent efforts in applying and adapting SAM2 to biomedical images and videos. The findings indicate that while SAM2 shows promise in reducing annotation burdens and enabling zero-shot segmentation, its performance varies across different datasets and tasks. Addressing the domain gap between natural and medical images through adaptation and fine-tuning is essential to fully unleash SAM2's potential in clinical applications. To support ongoing research endeavors, we maintain an active repository that contains up-to-date SAM & SAM2-related papers and projects at https://github.com/YichiZhang98/SAM4MIS.
From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究院）；School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；ByteDance China（字节跳动中国）；ByteDance USA（字节跳动美国）；Lingnan University（岭南大学）
Biomedical SAM 2: Segment Anything in Biomedical Images and Videos
University of Georgia（佐治亚大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Lehigh University（理海大学）；Northeastern University（东北大学）；University of Notre Dame（圣母大学）
TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Xiamen University（厦门大学）；Alibaba Group（阿里巴巴集团）
University of Notre Dame（圣母大学）；Nanjing University of Science and Technology（南京理工大学）；Leibniz-Institut für Analytische Wissenschaften–ISAS–e.V.（莱布尼茨分析科学研究所）
Deep learning (DL) based semantic segmentation methods have achieved excellent performance in biomedical image segmentation, producing high quality probability maps to allow extraction of rich instance information to facilitate good instance segmentation. While numerous efforts were put into developing new DL semantic segmentation models, less attention was paid to a key issue of how to effectively explore their probability maps to attain the best possible instance segmentation. We observe that probability maps by DL semantic segmentation models can be used to generate many possible instance c…
GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）；Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology（南京工程学院人工智能产业技术研究院）
Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Department of Computer Science and Engineering, University at Buffalo（计算机科学与工程系，布法罗大学）；Department of Surgery, University at Buffalo（外科系，布法罗大学）；Department of Computer Science and Engineering, IIT, Jodhpur（计算机科学与工程系，印度理工学院，乔杜尔）；Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，圣母大学）
University of Kaiserslautern（凯泽斯劳滕大学）；Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (The Swiss AI Lab IDSIA)（达莱莫勒人工智能研究所（瑞士人工智能实验室IDSIA））；Scuola universitaria professionale della Svizzera italiana (SUPSI)（瑞士意大利语区应用科学大学（SUPSI））；Universitá della Svizzera italiana (USI)（瑞士意大利语区大学（USI））；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Convolutional Neural Networks (CNNs) can be shifted across 2D images or 3D videos to segment them. They have a fixed input size and typically perceive only small local contexts of the pixels to be classified as foreground or background. In contrast, Multi-Dimensional Recurrent NNs (MD-RNNs) can perceive the entire spatio-temporal context of each pixel in a few sweeps through all pixels, especially when the RNN is a Long Short-Term Memory (LSTM). Despite these theoretical advantages, however, unlike CNNs, previous MD-LSTM variants were hard to parallelize on GPUs. Here we re-arrange the traditi…
Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
College of AI, Tsinghua University, Beijing, China（清华大学人工智能学院，北京，中国）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）；The University of Tokyo, Tokyo, Japan（东京大学，东京，日本）
How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?
Stanford University（斯坦福大学）；Together AI；University of California, Berkeley（加州大学伯克利分校）；Cedars-Sinai Medical Center（西德斯-西奈医学中心）；University of California, San Francisco（加州大学旧金山分校）
Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT（宁波空间智能与数字衍生关键实验室，数字孪生研究院，EIT）；Shanghai Jiao Tong University（上海交通大学）；Ocean University of China（中国海洋大学）
MoZoo:Unleashing Video Diffusion power in animal fur and muscle simulation
tabular c 1 Tsinghua University 2 University of Glasgow 3 The Chinese University of Hong Kong 4 HUIJING Digital Media \& Entertainment Group 2mm Project Page: tabular
Bora: Biomedical Generalist Video Generation Model
University of Notre Dame（圣母大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Lehigh University（理海大学）；Northeastern University（东北大学）；Shanghai University of Finance and Economics（上海财经大学）；Fudan University（复旦大学）
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
The University of Hong Kong（香港大学）；ShanghaiTech University（上海科技大学）；Hong Kong University of Science and Technology（香港科学大学）；Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克研究所（信息学））；Texas A&M University（德克萨斯大学奥斯汀分校）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences University（中国科学院大学人工智能学院）；National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Center of Excellence in Brain Science and Intelligence Technology, CAS（中国科学院脑科学与智能技术卓越创新中心）
The continuity of biological tissue between consecutive biomedical images makes it possible for the video interpolation algorithm, to recover large area defects and tears that are common in biomedical images. However, noise and blur differences, large deformation, and drift between biomedical images, make the task challenging. To address the problem, this paper introduces a deformation-aware network to synthesize each pixel in accordance with the continuity of biological tissue. First, we develop a deformation-aware layer for consecutive biomedical images interpolation that implicitly adopting…
Recent advances in bioimaging have provided scientists a superior high spatial-temporal resolution to observe dynamics of living cells as 3D volumetric videos. Unfortunately, the 3D biomedical video analysis is lagging, impeded by resource insensitive human curation using off-the-shelf 3D analytic tools. Herein, biologists often need to discard a considerable amount of rich 3D spatial information by compromising on 2D analysis via maximum intensity projection. Recently, pixel embedding-based cell instance segmentation and tracking provided a neat and generalizable computing paradigm for unders…
École Centrale de Lyon（里昂中央理工学院）；University of Eastern Finland（东芬兰大学）
Spectral imaging has received enormous interest in the field of medical imaging modalities. It provides a powerful tool for the analysis of different organs and non-invasive tissues. Therefore, significant amount of research has been conducted to explore the possibility of using spectral imaging in biomedical applications. To observe spectral image information in real time during surgery and monitor the temporal changes in the organs and tissues is a demanding task. Available spectral imaging devices are not sufficient to accomplish this task with an acceptable spatial and spectral resolution.…
Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根萨大学医学影像计算中心）；Research Group MITI, TUM University Hospital, School of Medicine and Health（技术大学慕尼黑大学医院MITI研究组）；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根萨生物医学工程研究中心）；Regensburg University（雷根萨大学）；Regensburg Center of Health Sciences and Technology (RCHST), OTH Regensburg（雷根萨健康科学与技术研究中心）；AI Centre of Excellence, Medtronic Ltd.（医学影像人工智能卓越中心）；Engineering Sciences, University College London（伦敦大学学院工程科学系）；Augmented Intelligence Lab, Kyung Hee University（庆熙大学增强智能实验室）；University of Minho, Braga（明霍大学）；Jmees Inc.（Jmees公司）；School of Sciences, São Paulo State University (UNESP), Bauru（圣保罗州立大学科学学院）；KIST HARILAB, Center for Humanoid Research, Artificial Intelligence and Robot Institute, Korea Institute of Science and Technology (KIST)（韩国科学技术院HARILAB中心）；King's College London（伦敦国王学院）；The Chinese University of Hong Kong（香港中文大学）；Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ)（德国癌症研究中心智能医疗系统部门）；Muroran Institute of Technology, Hokkaido（北海道Muroran技术学院）；Niigata University of Health and Welfare（Niigata医疗福利大学）；Konica Minolta, Inc.（东宝株式会社）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科技大学化学与生物工程系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Shenzhen（香港科技大学深圳-香港协同创新研究院）；Center for Research and Formation in Artificial Intelligence (CinfonIA), Los Andes University, Bogota（安第斯大学人工智能研究与培养中心）；Department of Biomedical Engineering, Medical Image Analysis Group, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）；Institute of Information Technology (ITEC), Klagenfurt University（克雷格夫大学信息技术研究所）；Center for Tactile Internet with Human-in-the-loop (CeTI), TU Dresden（德累斯顿技术大学触觉互联网中心）
Classifying the behavior of humans or animals from videos is important in biomedical fields for understanding brain function and response to stimuli. Action recognition, classifying activities performed by one or more subjects in a trimmed video, forms the basis of many of these techniques. Deep learning models for human action recognition have progressed significantly over the last decade. Recently, there is an increased interest in research that incorporates deep learning-based action recognition for animal behavior classification. However, human action recognition methods are more developed…
MedSAM3: Delving into Segment Anything with Medical Concepts
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Southeast University（东南大学）；The Hong Kong University of Science and Technology（香港科技大学）
