Current cross-modal retrieval systems are evaluated using R@K measure which does not leverage semantic relationships rather strictly follows the manually marked image text query pairs. Therefore, current systems do not generalize well for the unseen data in the wild. To handle this, we propose a new measure, SemanticMap, to evaluate the performance of cross-modal systems. Our proposed measure evaluates the semantic similarity between the image and text representations in the latent embedding space. We also propose a novel cross-modal retrieval system using a single stream network for bidirectional retrieval. The proposed system is based on a deep neural network trained using extended center loss, minimizing the distance of image and text descriptions in the latent space from the class centers. In our system, the text descriptions are also encoded as images which enabled us to use a single stream network for both text and images. To the best of our knowledge, our work is the first of its kind in terms of employing a single stream network for cross-modal retrieval systems. The proposed system is evaluated on two publicly available datasets including MSCOCO and Flickr30K and has shown comparable results to the current state-of-the-art methods.
CRCV, University of Central Florida(中佛罗里达大学CRCV);Goethe University Frankfurt Germany(法兰克福歌德大学);University of Bonn Germany(波恩大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning
The University of Texas Arlington(德克萨斯大学阿灵顿分校);RIKEN(理化学研究所);University of Tokyo(东京大学);National Cancer Center Research Institute(国立癌症中心研究所);National Library of Medicine - National Institutes of Health(美国国立卫生研究院国家医学图书馆);Temple University(天普大学)
End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
Beijing Institute of Technology(北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Shenzhen MSU-BIT University(深圳北理莫斯科大学);University of Rochester(罗切斯特大学)
Brain Tumor Segmentation Network Using Attention-based Fusion and Spatial Relationship Constraint
College of Physics and Information Engineering, Fuzhou University(福州大学物理与信息工程学院);School of Data Science, Fudan University(复旦大学数据科学学院);School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院);School of Biomedical Engineering and Imaging Sciences, King’s College London(伦敦国王学院生物医学工程与成像科学学院)
SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention
School of Automation Science and Engineering, South China University of Technology(华南理工大学自动化科学与工程学院);School of Software Engineering, South China University of Technology(华南理工大学软件学院)
BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation
Graduate School of Informatics, Nagoya University(名古屋大学信息学研究科);AI Thrust, The Hong Kong University of Science and Technology(香港科学与技术大学人工智能 thrust);INSAIT, Sofia University(索菲亚大学INSAIT)
MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations
Jilin University(吉林大学);Zhejiang University(浙江大学);Zhejiang Lab(之江实验室);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE(教育部知识驱动人机智能工程研究中心)
Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing
College of Computer Science and Electronic Engineering, Hunan University(计算机科学与电子工程学院,湖南大学);School of Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(机器人学院及机器人视觉感知与控制技术国家工程研究中心,湖南大学);School of Computing and Artificial Intelligence, Southwest Jiaotong University(计算与人工智能学院,西南交通大学)
CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data
College of Intelligence and Computing(智能与计算学院);Tianjin University(天津大学);Medical School of Tianjin University(天津大学医学院);Peng Cheng Lab(鹏城实验室);Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong(统计与精算系,计算与数据科学学院,香港大学);Department of Radiology, Tianjin Huanhu Hospital(天津华医院放射科);Tianjin Key Laboratory of Cerebral Vascular and Neurodegenerative Diseases(天津脑血管与神经退行性疾病重点实验室)
Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School(马萨诸塞州总医院与哈佛医学院戈登医学影像中心);Korea Advanced Institute of Science and Technology(韩国科学技术院)
Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(A*STAR));The Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学-新加坡南洋理工大学人工智能联合研究中心)
Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models
Department of Computer Science and Engineering, CUHK, Hong Kong, China(计算机科学与工程系,CUHK,香港,中国);Institute of Medical Intelligence and XR, CUHK, Hong Kong, China(医学智能与XR研究所,CUHK,香港,中国)
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
GFZ Helmholtz Center for Geosciences(德国波茨坦地学研究中心亥姆霍兹中心);INRAE, UMR TETIS, University of Montpellier(法国蒙彼利埃大学农业环境与生态研究院、蒙彼利埃大学TETIS联合研究单位);INRIA, EVERGREEN, University of Montpellier(法国蒙彼利埃大学信息与自动化研究所、EVERGREEN实验室);CIRAD, UMR TETIS, University of Montpellier(法国蒙彼利埃大学国际农业研究磋商组织、蒙彼利埃大学TETIS联合研究单位)
有效检测假新闻最近引起了极大的关注。目前的研究在预测假新闻方面做出了重大贡献,但较少关注利用新闻文章中文本和视觉信息之间的关系(相似性)。重视这种相似性有助于识别假新闻故事,例如,试图使用不相关的图像来吸引读者的注意力。在这项工作中,我们提出了一个$\mathsf{S}$ imilarity- $\mathsf{A}$ ware $\mathsf{F}$ ak $\mathsf{E}$新闻检测方法($\mathsf{SAFE}$),调查多模态(文本和视觉)信息的新闻文章。首先,采用神经网络分别提取新闻文本和视觉特征。我们进一步研究跨模态提取的特征之间的关系。新闻文本和视觉信息的这种表示及其关系被联合学习并用于预测假新闻。提出的方法有助于识别虚假的新闻文章的基础上,他们的文字,图像,或他们的“不匹配”。“我们对大规模的真实数据进行了广泛的实验,证明了所提出的方法的有效性。
IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection
College of Information Sciences and Technology(信息科学与技术学院);The Pennsylvania State University(宾夕法尼亚州立大学);Department of Computer Science and Engineering(计算机科学与工程系);National Institute of Technology, Tiruchirappalli(特里奇里帕利理工学院)