This paper aims to address universal segmentation for image and video perception with the strong reasoning ability empowered by Visual Large Language Models (VLLMs). Despite significant progress in current unified segmentation methods, limitations in adaptation to both image and video scenarios, as well as the complex reasoning segmentation, make it difficult for them to handle various challenging instructions and achieve an accurate understanding of fine-grained vision-language correlations. We propose HyperSeg, the first VLLM-based universal segmentation model for pixel-level image and video perception, encompassing generic segmentation tasks and more complex reasoning perception tasks requiring powerful reasoning abilities and world knowledge. Besides, to fully leverage the recognition capabilities of VLLMs and the fine-grained visual information, HyperSeg incorporates hybrid entity recognition and fine-grained visual perceiver modules for various segmentation tasks. Combined with the temporal adapter, HyperSeg achieves a comprehensive understanding of temporal information. Experimental results validate the effectiveness of our insights in resolving universal image and video segmentation tasks, including the more complex reasoning perception tasks. Our code is available.
DFormer: Diffusion-guided Transformer for Universal Image Segmentation
Tianjin University(天津大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Chongqing University(重庆大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Bosch Research North America(博世北美研究院);Bosch Center for Artificial Intelligence (BCAI)(博世人工智能中心);The Ohio State University(俄亥俄州立大学);University of California Davis(加州大学戴维斯分校)
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc(字节跳动公司);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
SegVol: Universal and Interactive Volumetric Medical Image Segmentation
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);BAAI(北京智源人工智能研究院);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学)
MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model
Northwestern Polytechnical University(西北工业大学);Zhejiang University(浙江大学);The University of Adelaide(阿德莱德大学);Australian Institute for Machine Learning (AIML)(澳大利亚机器学习研究所);Research & Development Institute of Northwestern Polytechnical University in Shenzhen(西北工业大学深圳研究院);Ningbo Institute of Northwestern Polytechnical University(西北工业大学宁波研究院)
MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation
Clinical Center(临床中心);National Center for Biotechnology Information(国家生物技术信息中心);National Library of Medicine(国家医学图书馆);National Institutes of Health(美国国立卫生研究院)
Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning
Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);University Hospital Tübingen(图宾根大学医院);German Center for Mental Health(德国心理健康中心)
Improved Baselines with Synchronized Encoding for Universal Medical Image Segmentation
Xi’an Jiaotong University(西安交通大学);School of Information and Communications Engineering(信息与通信工程学院);School of Mathematics(数学学院);School of Mathematics and Statistics(数学与统计学学院);Pazhou Laboratory (Huangpu)(黄埔实验室)
C^2GR: Coupled Comprehensive Generative Replay for a Continually Learnable Universal Segmentation Model
Shanghai Jiao Tong University(上海交通大学);Southeast University(东南大学);Stevens Institute of Technology(史蒂文斯理工学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Imperial College London(伦敦帝国学院)
LUMOS: Universal Semi-Supervised OCT Retinal Layer Segmentation with Hierarchical Reliable Mutual Learning
Southern University of Science and Technology(南方科技大学);Shenzhen Loop Area Institute(深圳河套学院);Jiaxing Research Institute, Southern University of Science and Technology(南方科技大学嘉兴研究院)
Learning Universal Shape Dictionary for Realtime Instance Segmentation
Shanghai Jiao Tong University(上海交通大学)
我们提出了一种新的明确的形状表示实例分割。根据对物体形状建模的不同,现有的实例分割系统可以分为隐式模型和显式模型。隐式方法,表示对象掩模/轮廓的棘手的网络参数,并产生它通过逐像素分类,是占主导地位的。然而,显式的方法,参数化的形状与简单的和可解释的模型,较少探索。由于生成最终形状的操作是轻量级的,显式方法比隐式方法具有明显的速度优势,这对于现实世界的应用至关重要。建议USD-Seg采用线性模型,稀疏编码与字典,对象的形状。首先,它从大量的形状数据集中学习字典,使任何形状都能够通过字典分解为线性组合。因此被称为“通用形状字典”。然后在普通目标检测器的基础上增加一个简单的形状向量回归头,使检测器以最小的开销获得分割能力。对于定量评估,我们使用平均精度(AP)和建议的效率AP(AP $_E$)度量,其目的也是衡量计算消耗的框架,以满足现实世界的应用程序的要求。我们报告了在具有挑战性的COCO数据集上的实验结果,其中我们在单个Titan Xp GPU上的单个模型在65 fps下以YOLOv 4作为基本检测器实现了35.8 AP和27.8 AP $_E$,在12 fps下以FCOS作为基本检测器实现了34.1 AP和28.6 AP $_E$。
Developing Foundation Models for Universal Segmentation from 3D Whole-Body Positron Emission Tomography
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China(复旦大学人工智能创新与孵化院,上海,中国);Shanghai Academy of Artificial Intelligence for Science, Shanghai, China(上海人工智能科学研究院,上海,中国);Department of Nuclear Medicine and PET Center, Huashan Hospital, Fudan University, Shanghai, China(复旦大学华山医院核医学科与PET中心);Human Phenome Institute, Fudan University, Shanghai, China(复旦大学人类表型研究院);Department of Nuclear Medicine, Zhejiang Cancer Hospital, Hangzhou Institute of Medicine, Chinese Academy of Sciences, Hangzhou, China(浙江省肿瘤医院核医学科,杭州医学院,中国科学院,杭州,中国);Shanghai Universal Medical Imaging Diagnostic Center, Shanghai, China(上海通用医学影像诊断中心,上海,中国);Hangzhou Universal Medical Imaging Diagnostic Center, Hangzhou, China(杭州通用医学影像诊断中心,杭州,中国);Department of Information and Intelligence Development, Zhongshan Hospital, Fudan University, Shanghai, China(复旦大学中山医院信息与智能发展科);Department of Nuclear Medicine, Inselspital, University of Bern, Switzerland(伯尔尼大学Inselspital核医学科,瑞士)
UCS: A Universal Model for Curvilinear Structure Segmentation
School of Computer Science and Technology, Wuhan University of Science and Technology(武汉理工大学计算机科学与技术学院);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology(湖北省智能信息处理与实时工业系统重点实验室);Institute for Infocomm Research (I 2 R), Agency for Science, Technology and Research (A*STAR)(信息与通信研究所以及科技研究局)
Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation
School of Biomedical Engineering, Division of Life Sciences;Medicine, University of Science;Technology of China (USTC), Hefei, Anhui 230026, China Center for Medical Imaging, Robotics, Analytic Computing \& Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu 215123, China Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China State Key Laboratory of Precision
LUMOS: Latent Universal Medical Priors for Segmentation
The University of Sydney, Sydney. NSW 2006, Australia Biological System \& Engineering Division, Lawrence Berkeley National Laboratory, Berkeley. CA 94720, USA Berkeley Biomedical Data Science Center, Lawrence Berkeley National Laboratory, Berkeley. CA 94720, USA
SAM-aware Test-time Adaptation for Universal Medical Image Segmentation
Department of Data Science & AI, Faculty of Information Technology, Monash University(数据科学与人工智能系,信息科技学院,莫纳什大学);Department of Computing and Department of Brain Sciences, Imperial College London(计算系和脑科学系,伦敦帝国学院);Mohamed bin Zayed University of Artificial Intelligence(莫卧儿·本·扎耶德人工智能大学);Department of Radiation Oncology, UT Southwestern Medical Center(放射肿瘤科,德克萨斯西南医学中心)