Semantic mapping based on the supervised object detectors is sensitive to image distribution. In real-world environments, the object detection and segmentation performance can lead to a major drop, preventing the use of semantic mapping in a wider domain. On the other hand, the development of vision-language foundation models demonstrates a strong zero-shot transferability across data distribution. It provides an opportunity to construct generalizable instance-aware semantic maps. Hence, this work explores how to boost instance-aware semantic mapping from object detection generated from foundation models. We propose a probabilistic label fusion method to predict close-set semantic classes from open-set label measurements. An instance refinement module merges the over-segmented instances caused by inconsistent segmentation. We integrate all the modules into a unified semantic mapping system. Reading a sequence of RGB-D input, our work incrementally reconstructs an instance-aware semantic map. We evaluate the zero-shot performance of our method in ScanNet and SceneNN datasets. Our method achieves 40.3 mean average precision (mAP) on the ScanNet semantic instance segmentation task. It outperforms the traditional semantic mapping method significantly.
Intelligent Communication Mixture-of-Experts Boosted-Medical Image Segmentation Foundation Model
College of Optoelectronics Engineering, Chongqing University, Chongqing, China；Center of Digital Dentistry, Peking University School；Hospital of Stomatology \& NHC Key Laboratory of Digital Stomatology, Beijing, China
This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task of inferring the occupancy and semantic labels of a volumetric 3D scene from single-view RGB-D images. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously via leveraging the experience of inferring 2D semantic segmentation from RGB-D images as well as the reliable depth cues in spati…
College of Artificial Intelligence, Xi’an Jiaotong University（西安交通大学人工智能学院）；Nanjing University of Science and Technology（南京理工大学）；Hikvision Research America（海康威视美国研究院）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）
In this paper, we propose a Multi-Scale Boosted Dehazing Network with Dense Feature Fusion based on the U-Net architecture. The proposed method is designed based on two principles, boosting and error feedback, and we show that they are suitable for the dehazing problem. By incorporating the Strengthen-Operate-Subtract boosting strategy in the decoder of the proposed model, we develop a simple yet effective boosted decoder to progressively restore the haze-free image. To address the issue of preserving spatial information in the U-Net architecture, we design a dense feature fusion module using …
DinoRADE: Full Spectral Radar-Camera Fusion with Vision Foundation Model Features for Multi-class Object Detection in Adverse Weather
Infineon Technologies AG, Austria（英飞凌科技公司，奥地利）；Graz University of Technology, Austria（格拉茨技术大学，奥地利）；Virtual Vehicle Research GmbH, Austria（虚拟车辆研究有限公司，奥地利）
Northwestern University（西北大学）；University of Chicago（芝加哥大学）
Accurate segmentation of organs-at-risks (OARs) is a precursor for optimizing radiation therapy planning. Existing deep learning-based multi-scale fusion architectures have demonstrated a tremendous capacity for 2D medical image segmentation. The key to their success is aggregating global context and maintaining high resolution representations. However, when translated into 3D segmentation problems, existing multi-scale fusion architectures might underperform due to their heavy computation overhead and substantial data diet. To address this issue, we propose a new OAR segmentation framework, c…
Feature pyramid networks (FPN) are widely exploited for multi-scale feature fusion in existing advanced object detection frameworks. Numerous previous works have developed various structures for bidirectional feature fusion, all of which are shown to improve the detection performance effectively. We observe that these complicated network structures require feature pyramids to be stacked in a fixed order, which introduces longer pipelines and reduces the inference speed. Moreover, semantics from non-adjacent levels are diluted in the feature pyramid since only features at adjacent pyramid level…
SemiSAM-O1: Pushing the Boundary of Annotation-Efficient Medical Image Segmentation with Generalist Knowledge Fusion
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China.（复旦大学人工智能创新与孵化院，上海，中国）；Shanghai Academy of Artificial Intelligence for Science, Shanghai, China.（上海人工智能科学研究院，上海，中国）；Department of Radiotherapy, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China.（同济大学附属同济医院放疗科，上海，中国）；School of Information Science and Engineering, Lanzhou University, Lanzhou, China.（兰州大学信息科学与工程学院，兰州，中国）；Department of Information and Intelligence Development, Zhongshan Hospital, Fudan University, Shanghai, China.（复旦大学中山医院信息与智能发展科，上海，中国）
Foundation-Model-Boosted Multimodal Learning for fMRI-based Neuropathic Pain Drug Response Prediction
School of Computer Science, University of Sheffield（谢菲尔德大学计算机学院）；Centre for Machine Intelligence, University of Sheffield（谢菲尔德大学机器智能中心）；Academic Unit of Radiology, University of Sheffield（谢菲尔德大学放射学学术单元）；Diabetes Research Unit, Royal Hallamshire Hospital（皇家哈勒姆郡医院糖尿病研究单元）；Division of Clinical Medicine, University of Sheffield（谢菲尔德大学临床医学部）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）
Fusion of Foundation and Vision Transformer Model Features for Dermatoscopic Image Classification
Research Center for Medical Image Analysis and Artificial Intelligence, Department of Medicine, Faculty of Medicine and Dentistry, Danube Private University（医学影像分析与人工智能研究中心、医学部、医学与牙科学院、多瑙私人大学）；Department of Research and Development, TissueGnostics GmbH（研发部、TissueGnostics GmbH）
Radiomics--Foundation Fusion for Interpretable RCC Classification: Internal Benchmarking and Exploratory External Transfer
Research Ireland Centre for Research Training in Machine Learning（爱尔兰研究机构机器学习研究培训中心）；University College Dublin（都柏林大学学院）；School of Veterinary Medicine, University College Dublin（都柏林大学学院兽医学院）
Shenzhen University（深圳大学）；The Ohio State University（俄亥俄州立大学）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）
In recent years, advances in the development of whole-slide images have laid a foundation for the utilization of digital images in pathology. With the assistance of computer images analysis that automatically identifies tissue or cell types, they have greatly improved the histopathologic interpretation and diagnosis accuracy. In this paper, the Convolutional Neutral Network (CNN) has been adapted to predict and classify lymph node metastasis in breast cancer. Unlike traditional image cropping methods that are only suitable for large resolution images, we propose a novel data augmentation metho…
DBCF: Dual-Branch Complementary Fusion of Foundation Models for Generalized Deepfake Detection
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Ocean University of China（中国海洋大学）
Understanding Synergistic Interactions among Pathology Foundation Models via Adaptive Fusion
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）；Leicester Cancer Research Centre, University of Leicester（莱斯特大学莱斯特癌症研究中心）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Nuffield Department of Medicine, University of Oxford（牛津大学纳菲尔德医学院）；A*STAR, Singapore（新加坡科技研究局）
Information-driven Fusion of Pathology Foundation Models for Enhanced Disease Characterization
Case Western Reserve University（凯斯西储大学）；Department of Biomedical Engineering（生物医学工程系）；Cleveland Clinic（克利夫兰诊所）；Department of Pathology（病理学系）；Emory University（埃默里大学）；Department of Pediatrics（儿科学系）；Louis Stokes VA Cleveland Medical Center（路易斯·斯托克斯退伍军人医疗中心）
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
CB-HVTNet: A channel-boosted hybrid vision transformer network for lymphocyte assessment in histopathological images
Pakistan Institute of Engineering & Applied Sciences（巴基斯坦工程与应用科学学院）；Air University（空军大学）；Universiti Teknologi PETRONAS（马来西亚国油科技大学）；Korea National University of Transportation（韩国交通大学）
FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
Giessen University（吉森大学）；University Medicine Gottingen（哥廷根大学医学中心）；Technical University of Denmark（丹麦技术大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Max Planck Institute for Biology of Ageing（马克斯·普朗克衰老生物学研究所）；National Taiwan University（国立台湾大学）；VNU University of Medicine and Pharmacy（越南国立大学医学与药学大学）；University of Medicine and Pharmacy, Hue University（顺化大学医学与药学大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Stuttgart（斯图加特大学）
Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion
National-Local Joint Engineering Research Center of Biodiagnosis \& Biotherapy, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi’an, 710004, China Institute of Medical Artificial Intelligence, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi'an, 710004, China Viadrina European University, Frankfurt (Oder), 15230, Germany
Attentive multilayer fusion for vision transformers
Hector Fellow Academy（海克荣誉院士协会）；BIFOLD；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Helmholtz Munich（亥姆霍兹慕尼黑）；Technical University Munich（慕尼黑技术大学）；MCML（慕尼黑机器学习中心）；Aignostics, Berlin（柏林Aignostics）；RIKEN AIP（理化学研究所AIP）
Fusion of Multi-scale Heterogeneous Pathology Foundation Models for Whole Slide Image Analysis
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学生物医学工程学院）；Division of Life Science, Department of Chemical and Biological Engineering, State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology（香港科技大学生命科学系）；SIAT-HKUST Joint Laboratory of Cell Evolution and Digital Health, HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（深圳-香港联合创新研究院细胞进化与数字健康联合实验室）；Department of Hepatobiliary Surgery, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学附属第一医院肝胆外科）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China（中国科学技术大学苏州先进研究所）；Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou, Jiangsu, China（江苏省多模态数字孪生技术重点实验室）；Key Laboratory of Precision and Intelligent Chemistry, USTC, Hefei, Anhui, China（中国科学技术大学精准与智能化学重点实验室）；Department of Pathology, Chinese PLA General Hospital, Beijing, China（中国人民解放军总医院病理科）
Carnegie Mellon University（卡内基梅隆大学）；The Bronx High School of Science（布朗克斯科学高中）
Reliable curb detection is critical for safe autonomous driving in urban contexts. Curb detection and tracking are also useful in vehicle localization and path planning. Past work utilized a 3D LiDAR sensor to determine accurate distance information and the geometric attributes of curbs. However, such an approach requires dense point cloud data and is also vulnerable to false positives from obstacles present on both road and off-road areas. In this paper, we propose an approach to detect and track curbs by fusing together data from multiple sensors: sparse LiDAR data, a mono camera and low-cos…
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a "foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision …
Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion
College of Computer Science, Chongqing University（重庆大学计算机学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
