3D patient body modeling is critical to the success of automated patient positioning for smart medical scanning and operating rooms. Existing CNN-based end-to-end patient modeling solutions typically require a) customized network designs demanding large amount of relevant training data, covering extensive realistic clinical scenarios (e.g., patient covered by sheets), which leads to suboptimal generalizability in practical deployment, b) expensive 3D human model annotations, i.e., requiring huge amount of manual effort, resulting in systems that scale poorly. To address these issues, we propose a generic modularized 3D patient modeling method consists of (a) a multi-modal keypoint detection module with attentive fusion for 2D patient joint localization, to learn complementary cross-modality patient body information, leading to improved keypoint localization robustness and generalizability in a wide variety of imaging (e.g., CT, MRI etc.) and clinical scenarios (e.g., heavy occlusions); and (b) a self-supervised 3D mesh regression module which does not require expensive 3D mesh parameter annotations to train, bringing immediate cost benefits for clinical deployment. We demonstrate the efficacy of the proposed method by extensive patient positioning experiments on both public and clinical data. Our evaluation results achieve superior patient positioning performance across various imaging modalities in real clinical scenarios.
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement
State Key Laboratory of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；National Computer Network Emergency Response Technical Team, Coordination Center of China（中国国家计算机网络应急技术处理协调中心）
An Interpretable Cross-Attentive Multi-modal MRI Fusion Framework for Schizophrenia Diagnosis
Tulane University（杜兰大学）；Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（跨机构神经成像与数据科学转化研究中心（TReNDS））；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
Despite the successes of deep neural networks on many challenging vision tasks, they often fail to generalize to new test domains that are not distributed identically to the training data. The domain adaptation becomes more challenging for cross-modality medical data with a notable domain shift. Given that specific annotated imaging modalities may not be accessible nor complete. Our proposed solution is based on the cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists and bridge the domain gap in radiological images. We present a novel approach for …
We propose cross-modal attentive connections, a new dynamic and effective technique for multimodal representation learning from wearable data. Our solution can be integrated into any stage of the pipeline, i.e., after any convolutional layer or block, to create intermediate connections between individual streams responsible for processing each modality. Additionally, our method benefits from two properties. First, it can share information uni-directionally (from one modality to the other) or bi-directionally. Second, it can be integrated into multiple stages at the same time to further allow n…
Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection
Department of Computer Science and Engineering, Bangladesh University of Business and Technology（计算机科学与工程系，孟加拉国商业技术大学）；Department of Computer Science, American International University-Bangladesh（计算机科学系，美国国际大学-孟加拉国）
Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）
We propose an audio-visual spatial-temporal deep neural network with: (1) a visual block containing a pretrained 2D-CNN followed by a temporal convolutional network (TCN); (2) an aural block containing several parallel TCNs; and (3) a leader-follower attentive fusion block combining the audio-visual information. The TCN with large history coverage enables our model to exploit spatial-temporal information within a much larger window length (i.e., 300) than that from the baseline and state-of-the-art methods (i.e., 36 or 48). The fusion block emphasizes the visual modality while exploits the noi…
Wenzhou University（温州大学）；University of Manitoba（曼尼托巴大学）；Shanghai Institute for Advanced Communication and Data Science（上海先进通信与数据科学研究院）；Shanghai University（上海大学）
We consider the problem of referring segmentation in images and videos with natural language. Given an input image (or video) and a referring expression, the goal is to segment the entity referred by the expression in the image or video. In this paper, we propose a cross-modal self-attention (CMSA) module to utilize fine details of individual words and the input image or video, which effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the visual input. W…
University of Washington（华盛顿大学）；Texas A&M University（德克萨斯A&M大学）
We present our preliminary work to determine if patient's vocal acoustic, linguistic, and facial patterns could predict clinical ratings of depression severity, namely Patient Health Questionnaire depression scale (PHQ-8). We proposed a multi modal fusion model that combines three different modalities: audio, video , and text features. By training over AVEC 2017 data set, our proposed model outperforms each single modality prediction model, and surpasses the data set baseline with ice margin.
Nanjing University of Science and Technology（南京理工大学）
This work focuses on the task of elderly activity recognition, which is a challenging task due to the existence of individual actions and human-object interactions in elderly activities. Thus, we attempt to effectively aggregate the discriminative information of actions and interactions from both RGB videos and skeleton sequences by attentively fusing multi-modal features. Recently, some nonlinear multi-modal fusion approaches are proposed by utilizing nonlinear attention mechanism that is extended from Squeeze-and-Excitation Networks (SENet). Inspired by this, we propose a novel Expansion-Squ…
Current LiDAR-only 3D detection methods inevitably suffer from the sparsity of point clouds. Many multi-modal methods are proposed to alleviate this issue, while different representations of images and point clouds make it difficult to fuse them, resulting in suboptimal performance. In this paper, we present a novel multi-modal framework SFD (Sparse Fuse Dense), which utilizes pseudo point clouds generated from depth completion to tackle the issues mentioned above. Different from prior works, we propose a new RoI fusion strategy 3D-GAF (3D Grid-wise Attentive Fusion) to make fuller use of info…
The use of multi-modal data for deep machine learning has shown promise when compared to uni-modal approaches with fusion of multi-modal features resulting in improved performance in several applications. However, most state-of-the-art methods use naive fusion which processes feature streams independently, ignoring possible long-term dependencies within the data during fusion. In this paper, we present a novel Memory based Attentive Fusion layer, which fuses modes by incorporating both the current features and longterm dependencies in the data, thus allowing the model to understand the relativ…
Universitá degli Studi di Trento（特伦托大学）；Universitá degli Studi di Modena e Reggio Emilia（摩德纳-雷焦艾米利亚大学）；Universitá degli Studi di Padova（帕多瓦大学）
Anticipating human motion in crowded scenarios is essential for developing intelligent transportation systems, social-aware robots and advanced video surveillance applications. A key component of this task is represented by the inherently multi-modal nature of human paths which makes socially acceptable multiple futures when human interactions are involved. To this end, we propose a generative architecture for multi-future trajectory predictions based on Conditional Variational Recurrent Neural Networks (C-VRNNs). Conditioning mainly relies on prior belief maps, representing most likely moving…
Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives, as well as the availability of vast amounts of digital cooking recipes and food images to train machine learning models. In this work, we revisit existing approaches for cross-modal recipe retrieval and propose a simplified end-to-end model based on well established and high performing encoders for text and images. We introduce a hierarchical recipe Transformer which attentively encodes individual recipe components (titles, ingredients and instructions). Further, we propose a…
The automatic diagnosis of various retinal diseases from fundus images is important to support clinical decision-making. However, developing such automatic solutions is challenging due to the requirement of a large amount of human-annotated data. Recently, unsupervised/self-supervised feature learning techniques receive a lot of attention, as they do not need massive annotations. Most of the current self-supervised methods are analyzed with single imaging modality and there is no method currently utilize multi-modal images for better results. Considering that the diagnostics of various vitreor…
University of Maryland（马里兰大学）；PAII Inc.（PAII公司）；Alibaba Group DAMO Academy（阿里巴巴集团达摩院）；Xpeng Motors（小鹏汽车）；Ping An Technology（平安科技）；Chang Gung Memorial Hospital（长庚纪念医院）
Osteoporosis is a common chronic metabolic bone disease often under-diagnosed and under-treated due to the limited access to bone mineral density (BMD) examinations, e.g. via Dual-energy X-ray Absorptiometry (DXA). This paper proposes a method to predict BMD from Chest X-ray (CXR), one of the most commonly accessible and low-cost medical imaging examinations. Our method first automatically detects Regions of Interest (ROIs) of local CXR bone structures. Then a multi-ROI deep model with transformer encoder is developed to exploit both local and global information in the chest X-ray image for ac…
Multi-modal 3D Pose and Shape Estimation with Computed Tomography
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学计算机科学学院）；School of Biomedical Engineering, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学生物医学工程学院）；Trauma Service, Westmead Hospital, Australia（西墨迪医院创伤服务）
Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers
Heidelberg University Hospital（海德堡大学医院）；Informatics for Life Institute（生命信息学研究所）；German Centre for Cardiovascular Research（德国心血管研究中心）；University Medical Center of the Johannes Gutenberg University Mainz（美因茨约翰内斯·古滕贝格大学医学中心）；University Medical Center Mainz（美因茨大学医学中心）；Institute of Molecular Biology (IMB)（分子生物学研究所）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））
Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the inner-modal attentive weights may also be diluted, which could thus undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and subst…
Routine clinical visits of a patient produce not only image data, but also non-image data containing clinical information regarding the patient, i.e., medical data is multi-modal in nature. Such heterogeneous modalities offer different and complementary perspectives on the same patient, resulting in more accurate clinical decisions when they are properly combined. However, despite its significance, how to effectively fuse the multi-modal medical data into a unified framework has received relatively little attention. In this paper, we propose an effective graph-based framework called HetMed (He…
Modality-agnostic, patient-specific digital twins modeling temporally varying digestive motion
Computer and Information Science, Cornell University（康奈尔大学计算机与信息科学系）；Department of Medical Physics, Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心医学物理系）；University Medical Center Utrecht（乌得勒支大学医学中心）；Department of Radiation Physics, University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心放射物理系）；Carl E. Ravin Advanced Imaging Laboratories and Center for Virtual Imaging Trials, Duke University Medical Center（杜克大学医学中心卡尔·E·拉文高级影像实验室和虚拟影像试验中心）；Department of Radiation Oncology, University of Michigan（密歇根大学放射肿瘤学系）
University of Padova（帕多瓦大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Human trajectory forecasting is a key component of autonomous vehicles, social-aware robots and advanced video-surveillance applications. This challenging task typically requires knowledge about past motion, the environment and likely destination areas. In this context, multi-modality is a fundamental aspect and its effective modeling can be beneficial to any architecture. Inferring accurate trajectories is nevertheless challenging, due to the inherently uncertain nature of the future. To overcome these difficulties, recent models use different inputs and propose to model human intentions usin…
Inception Institute of Artificial Intelligence（人工智能起源研究所）；Lehigh University（利哈伊大学）；Tianjin University（天津大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not represent the full image or model complex tumor patterns. Second, different types of scanners (i.e., multi-modal data) are sensitive to different brain regions, which makes it challenging to effectively ex…
University of Manitoba（曼尼托巴大学）；Shanghai University（上海大学）
We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on info…
ARC Training Centre for Innovative BioEngineering（ARC创新生物工程培训中心）
Distant metastases (DM) refer to the dissemination of tumors, usually, beyond the organ where the tumor originated. They are the leading cause of death in patients with soft-tissue sarcomas (STSs). Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of STSs. It is difficult to determine from imaging studies which STS patients will develop metastases. 'Radiomics' refers to the extraction and analysis of quantitative features from medical images and it has been employed to help identify such tumors. The state-of-the-art in ra…
University of Oxford（牛津大学）；Fuzhou University（福州大学）；Fudan University（复旦大学）
Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optima…
Prediction of Distant Metastasis in Head and Neck Cancer Patients Using Tumor and Peritumoral Multi-Modal Deep Learning
Xidian University（西安电子科技大学）；Xijing Hospital, Air Force Medical University of PLA（中国人民解放军空军军医大学西京医院）；The General Hospital of Western Theater Command（西部战区总医院）
Queen’s University（女王大学）；Azad University（阿扎德大学）；University of Leeds（利兹大学）；KU Leuven（荷语鲁汶大学）；Shenzhen University（深圳大学）
In large studies involving multi protocol Magnetic Resonance Imaging (MRI), it can occur to miss one or more sub-modalities for a given patient owing to poor quality (e.g. imaging artifacts), failed acquisitions, or hallway interrupted imaging examinations. In some cases, certain protocols are unavailable due to limited scan time or to retrospectively harmonise the imaging protocols of two independent studies. Missing image modalities pose a challenge to segmentation frameworks as complementary information contributed by the missing scans is then lost. In this paper, we propose a novel model, …
MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training
The Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen People’s Hospital（深圳市人民医院）；The University of Hong Kong（香港大学）；Chinese PLA General Hospital（中国人民解放军总医院）
HIST-AID: Leveraging Historical Patient Reports for Enhanced Multi-Modal Automatic Diagnosis
Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；New York University Grossman School of Medicine（纽约大学格罗斯曼医学院）；Center of Data Science, New York University（纽约大学数据科学中心）；Prescient Design, Genentech（基因泰克公司Prescient Design部门）；CIFAR LMB（加拿大高级研究院学习算法与机器智能实验室）
The explosive rise of the use of Computer tomography (CT) imaging in medical practice has heightened public concern over the patient's associated radiation dose. However, reducing the radiation dose leads to increased noise and artifacts, which adversely degrades the scan's interpretability. Consequently, an advanced image reconstruction algorithm to improve the diagnostic performance of low dose ct arose as the primary concern among the researchers, which is challenging due to the ill-posedness of the problem. In recent times, the deep learning-based technique has emerged as a dominant method…
Inception Institute of Artificial Intelligence（人工智能 inception 研究所）
Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution to this, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-mo…
MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（高等技术学院）；Institute of Computing（计算机研究所）；University of Campinas（坎皮纳斯大学）；Lisbon ELLIS Unit（里斯本ELLIS单元）
Rethinking Patient Education as Multi-turn Multi-modal Interaction
VA Bedford Health Care（VA贝德福德医疗中心）；UMass Amherst（马萨诸塞大学阿默斯特分校）；UMass Lowell（马萨诸塞大学洛厄尔分校）；Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）；Yale School of Medicine（耶鲁医学院）
Northwestern University（西北大学）；Northwestern Memorial Hospital（西北纪念医院）
Propose: To present DeepCOVID-Fuse, a deep learning fusion model to predict risk levels in patients with confirmed coronavirus disease 2019 (COVID-19) and to evaluate the performance of pre-trained fusion models on full or partial combination of chest x-ray (CXRs) or chest radiograph and clinical variables. Materials and Methods: The initial CXRs, clinical variables and outcomes (i.e., mortality, intubation, hospital length of stay, ICU admission) were collected from February 2020 to April 2020 with reverse-transcription polymerase chain reaction (RT-PCR) test results as the reference standard…
Improving Out-of-distribution Human Activity Recognition via IMU-Video Cross-modal Representation Learning
Department of Computer Science & Engineering University of Minnesota（计算机科学与工程系明尼苏达大学）；Department of Mechanical Engineering University of Minnesota（机械工程系明尼苏达大学）；Department of Neurosurgery University of Minnesota（神经外科系明尼苏达大学）
Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School（戈登医学影像中心，麻省总医院与哈佛医学院）；University of Maryland School of Medicine（马里兰大学医学院）；University of Maryland School of Dentistry（马里兰大学牙医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Yale University（耶鲁大学）
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Fundacion Progreso y Salud（进步与健康基金会）；National Tsing Hua University（台湾清华大学）；Albert Einstein Medical Center（阿尔伯特·爱因斯坦医疗中心）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）；National University Hospital（新加坡国立大学医院）；Hoboken University Medical Center–CarePoint Health（霍博肯大学医疗中心-CarePoint Health）；Hospital Universitario Monteprincipe（蒙特普林西比大学医院）；Grupo HM Hospitales（HM医院集团）；Seoul National University College of Medicine（首尔大学医学院）；Centro Nacional de Investigaciones Cardiovasculares（国家心血管研究中心）；Instituto de Salud Carlos III（卡洛斯三世健康研究所）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）
The unprecedented global crisis brought about by the COVID-19 pandemic has sparked numerous efforts to create predictive models for the detection and prognostication of SARS-CoV-2 infections with the goal of helping health systems allocate resources. Machine learning models, in particular, hold promise for their ability to leverage patient clinical information and medical images for prediction. However, most of the published COVID-19 prediction models thus far have little clinical utility due to methodological flaws and lack of appropriate validation. In this paper, we describe our methodology…
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Multi-modal neuroimaging technology has greatlly facilitated the efficiency and diagnosis accuracy, which provides complementary information in discovering objective disease biomarkers. Conventional deep learning methods, e.g. convolutional neural networks, overlook relationships between nodes and fail to capture topological properties in graphs. Graph neural networks have been proven to be of great importance in modeling brain connectome networks and relating disease-specific patterns. However, most existing graph methods explicitly require known graph structures, which are not available in t…
A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models
Junia（朱尼亚高等工程学院）；University of Lille（里尔大学）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；IMT Nord Europe（北欧电信高等学院）；Institut Mines-Télécom（法国国立矿业与电信学院）；University Hospital of Lille（里尔大学医院）；The University of Tokyo（东京大学）；Yncrea Méditerranée（Yncrea地中海分部）；Yncrea Ouest（Yncrea西部分部）
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Baptist University（香港浸会大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
A Multi-Modal Deep Learning Framework for Pan-Cancer Prognosis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Beijing Key Laboratory of Network System and Network Culture（网络系统与网络文化北京市重点实验室）；School of Basic Medical Sciences, Third Hospital, Peking University（北京大学第三医院基础医学院）
