Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cross- and intra-modal knowledge in a unified scene graph to enhance the semantic alignments. Specifically, we introduce a novel structural knowledge masking (SKM) strategy to use the scene graph structure as a priori to perform masked language (region) modeling, which enhances the semantic alignments by eliminating the interference information within and across modalities. Extensive ablation studies and comprehensive analysis verifies the effectiveness of ROSITA in semantic alignments. Pretrained with both in-domain and out-of-domain datasets, ROSITA significantly outperforms existing state-of-the-art VLP methods on three typical vision-and-language tasks over six benchmark datasets.
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Media Integration and Communication Center (MICC), University of Florence, Italy（意大利佛罗伦萨大学媒体集成与通信中心）；Department of Computer Science, Universitat Autònoma de Barcelona, Spain（西班牙巴塞罗那自治大学计算机科学系）；Computer Vision Center, Barcelona, Spain（西班牙巴塞罗那计算机视觉中心）；IDEAS Research Institute, Warsaw, Poland（波兰华沙IDEAS研究所）
Up to now, only limited research has been conducted on cross-modal retrieval of suitable music for a specified video or vice versa. Moreover, much of the existing research relies on metadata such as keywords, tags, or associated description that must be individually produced and attached posterior. This paper introduces a new content-based, cross-modal retrieval method for video and music that is implemented through deep neural networks. We train the network via inter-modal ranking loss such that videos and music with similar semantics end up close together in the embedding space. However, if …
Northwestern Polytechnical University（西北工业大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能有限公司）；Shanghai Jiao Tong University（上海交通大学）
Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-modality image registration network, which can directly predict the transformation field from the input multimodal images, such as CT and MR images. In particular, the training of our inter-modality registration network is supervised by intra-modality similarity metric based on the available paired data, which is derived from a pre-aligned CT and MR da…
Sichuan University（四川大学）；University of Electronic Science and Technology of China（电子科技大学）；Dalian University of Technology（大连理工大学）；National University of Defense Technology（国防科技大学）
Domain adaptation is critical for success when confronting with the lack of annotations in a new domain. As the huge time consumption of labeling process on 3D point cloud, domain adaptation for 3D semantic segmentation is of great expectation. With the rise of multi-modal datasets, large amount of 2D images are accessible besides 3D point clouds. In light of this, we propose to further leverage 2D data for 3D domain adaptation by intra and inter domain cross modal learning. As for intra-domain cross modal learning, most existing works sample the dense 2D pixel-wise features into the same size…
MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
Fudan University（复旦大学）；Shanghai Innovation Institut（上海创新研究院）；University of Southern California（南加州大学）；Huawei Technologies Co., Ltd（华为技术有限公司）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT（宁波空间智能与数字衍生关键实验室，数字孪生研究院，EIT）
Visible thermal person re-identification (VT-ReID) suffers from the inter-modality discrepancy and intra-identity variations. Distribution alignment is a popular solution for VT-ReID, which, however, is usually restricted to the influence of the intra-identity variations. In this paper, we propose the Cross-Modality Earth Mover's Distance (CM-EMD) that can alleviate the impact of the intra-identity variations during modality alignment. CM-EMD selects an optimal transport strategy and assigns high weights to pairs that have a smaller intra-identity variation. In this manner, the model will focu…
SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；HAOMO.AI Technology Co., Ltd.（毫末智行科技有限公司）；Zhejiang GongShang University（浙江工商大学）
Univ Rennes（雷恩大学）；INSA Rennes（雷恩国立应用科学学院）；CNRS（法国国家科学研究中心）；Zhejiang University of Technology（浙江工业大学）
Cross-modal retrieval has drawn much attention in both computer vision and natural language processing domains. With the development of convolutional and recurrent neural networks, the bottleneck of retrieval across image-text modalities is no longer the extraction of image and text features but an efficient loss function learning in embedding space. Many loss functions try to closer pairwise features from heterogeneous modalities. This paper proposes a method for learning joint embedding of images and texts using an intra-modal constraint loss function to reduce the violation of negative pair…
University of Science and Technology of China（中国科学技术大学）；The University of Adelaide（阿德莱德大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Shanghai Artificial Intelligent Laboratory（上海人工智能实验室）
Recent progresses on self-supervised 3D human action representation learning are largely attributed to contrastive learning. However, in conventional contrastive frameworks, the rich complementarity between different skeleton modalities remains under-explored. Moreover, optimized with distinguishing self-augmented samples, models struggle with numerous similar positive instances in the case of limited action categories. In this work, we tackle the aforementioned problems by introducing a general Inter- and Intra-modal Mutual Distillation (I$^2$MD) framework. In I$^2$MD, we first re-formulate t…
Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel…
RGB-infrared person re-identification is a challenging task due to the intra-class variations and cross-modality discrepancy. Existing works mainly focus on learning modality-shared global representations by aligning image styles or feature distributions across modalities, while local feature from body part and relationships between person images are largely neglected. In this paper, we propose a Dual-level (i.e., local and global) Feature Fusion (DF^2) module by learning attention for discriminative feature from local to global manner. In particular, the attention for a local feature is deter…
RGB-Infrared (IR) cross-modality person re-identification (re-ID), which aims to search an IR image in RGB gallery or vice versa, is a challenging task due to the large discrepancy between IR and RGB modalities. Existing methods address this challenge typically by aligning feature distributions or image styles across modalities, whereas the very useful similarities among gallery samples of the same modality (i.e. intra-modality sample similarities) is largely neglected. This paper presents a novel similarity inference metric (SIM) that exploits the intra-modality sample similarities to circumv…
University of Southern California（南加州大学）；Adobe Research（Adobe研究院）
Given a user's query, traditional image search systems rank images according to its relevance to a single modality (e.g., image content or surrounding text). Nowadays, an increasing number of images on the Internet are available with associated meta data in rich modalities (e.g., titles, keywords, tags, etc.), which can be exploited for better similarity measure with queries. In this paper, we leverage visual and textual modalities for image search by learning their correlation with input query. According to the intent of query, attention mechanism can be introduced to adaptively balance the i…
School of Computer and Information Technology at Beijing Jiaotong University（北京交通大学计算机与信息技术学院）
With the aim of matching a pair of instances from two different modalities, cross modality mapping has attracted growing attention in the computer vision community. Existing methods usually formulate the mapping function as the similarity measure between the pair of instance features, which are embedded to a common space. However, we observe that the relationships among the instances within a single modality (intra relations) and those between the pair of heterogeneous instances (inter relations) are insufficiently explored in previous approaches. Motivated by this, we redefine the mapping fun…
Istanbul Technical University（伊斯坦布尔理工大学）；Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）
Synthesizing multimodality medical data provides complementary knowledge and helps doctors make precise clinical decisions. Although promising, existing multimodal brain graph synthesis frameworks have several limitations. First, they mainly tackle only one problem (intra- or inter-modality), limiting their generalizability to synthesizing inter- and intra-modality simultaneously. Second, while few techniques work on super-resolving low-resolution brain graphs within a single modality (i.e., intra), inter-modality graph super-resolution remains unexplored though this would avoid the need for c…
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
School of Mechanical and Electric Engineering, Guangzhou University（广州大学机械与电气工程学院）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）
Cross-modality person re-identification is a challenging problem which retrieves a given pedestrian image in RGB modality among all the gallery images in infrared modality. The task can address the limitation of RGB-based person Re-ID in dark environments. Existing researches mainly focus on enlarging inter-class differences of feature to solve the problem. However, few studies investigate improving intra-class cross-modality similarity, which is important for this issue. In this paper, we propose a novel loss function, called Hetero-Center loss (HC loss) to reduce the intra-class cross-modali…
University of Bern（伯尔尼大学）；Inselspital（Inselspital医院（伯尔尼大学医院））；University Hospital of Fribourg（弗里堡大学医院）
Unsupervised domain adaptation (UDA) for cross-modality medical image segmentation has shown great progress by domain-invariant feature learning or image appearance translation. Adapted feature learning usually cannot detect domain shifts at the pixel level and is not able to achieve good results in dense semantic segmentation tasks. Image appearance translation, e.g. CycleGAN, translates images into different styles with good appearance, despite its population, its semantic consistency is hardly to maintain and results in poor cross-modality segmentation. In this paper, we propose intra- and …
Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Information Management, Jiangxi University of Finance and Economics（江西财经大学信息管理学院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究所）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
ICHPro: Intracerebral Hemorrhage Prognosis Classification Via Joint-attention Fusion-based 3d Cross-modal Network
Hangzhou Dianzi University（杭州电子科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Ping An Technology（平安科技）；Shenzhen University（深圳大学）；Longgang Central Hospital of Shenzhen（深圳市龙岗中心医院）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；Shenzhen Research Institute of Big Data（深圳市大数据研究院）
Sensetime Research（商汤科技研究院）；The Hong Kong Polytechnic University（香港理工大学）
Video highlight detection is a crucial yet challenging problem that aims to identify the interesting moments in untrimmed videos. The key to this task lies in effective video representations that jointly pursue two goals, \textit{i.e.}, cross-modal representation learning and fine-grained feature discrimination. In this paper, these two challenges are tackled by not only enriching intra-modality and cross-modality relations for representation modeling but also shaping the features in a discriminative manner. Our proposed method mainly leverages the intra-modality encoding and cross-modality co…
New York University（纽约大学）；University of Rochester（罗切斯特大学）；Siemens Healthineers（西门子医疗）
Using catheter ablation to treat atrial fibrillation increasingly relies on intracardiac echocardiography (ICE) for an anatomical delineation of the left atrium and the pulmonary veins that enter the atrium. However, it is a challenge to build an automatic contouring algorithm because ICE is noisy and provides only a limited 2D view of the 3D anatomy. This work provides the first automatic solution to segment the left atrium and the pulmonary veins from ICE. In this solution, we demonstrate the benefit of building a cross-modality framework that can leverage a database of diagnostic images to …
Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistencies
USC Information Sciences Institute（南加州大学信息科学研究所）；USC Ming Hsieh Department of Electrical and Computer Engineering（南加州大学明氏电气与计算机工程学院）；Clemson University（克莱姆森大学）；Holcombe Department of Electrical and Computer Engineering（霍尔科姆电气与计算机工程学院）
Nanjing University of Science and Technology（南京理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. Existing methods employ prior knowledge from single-modality pre-training to facilitate learning, but lack multi-modal correspondences. Besides, due to the substantial gap between modalities, existing methods embed the original modal features into the same latent space for cross-modal alignment. However, feature embedding may lead to intra-modal information distortion. Recently, CLIP has attracted extensive attention from researchers due to its powerful semantic concept learning capacity an…
School of Cyber Science and Technology, Beihang University（北京航空航天大学网络科学与技术学院）；College of Cyber Security, Jinan University（暨南大学网络安全学院）
Multimodal summarization with multimodal output (MSMO) generates a summary with both textual and visual content. Multimodal news report contains heterogeneous contents, which makes MSMO nontrivial. Moreover, it is observed that different modalities of data in the news report correlate hierarchically. Traditional MSMO methods indistinguishably handle different modalities of data by learning a representation for the whole data, which is not directly adaptable to the heterogeneous contents and hierarchical correlation. In this paper, we propose a hierarchical cross-modality semantic correlation l…
In this paper, we introduce Cross-modal Alignment with mixture experts Neural Network (CameNN) recommendation model for intral-city retail industry, which aims to provide fresh foods and groceries retailing within 5 hours delivery service arising for the outbreak of Coronavirus disease (COVID-19) pandemic around the world. We propose CameNN, which is a multi-task model with three tasks including Image to Text Alignment (ITA) task, Text to Image Alignment (TIA) task and CVR prediction task. We use pre-trained BERT to generate the text embedding and pre-trained InceptionV4 to generate image patc…
Inception Institute of Artificial Intelligence（起源人工智能研究所）；Indiana University（印第安纳大学）；University of Rochester（罗切斯特大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Due to the large intra-class variations and cross-modality discrepancy with large amount of sample noise, it is difficult to learn discriminative part features. Existing VI-ReID methods instead tend to learn global representations, which have limited discriminability and weak robustness to noisy images. In this paper, we propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues f…
Singapore University of Technology and Design（新加坡科技设计大学）；Monash University（蒙纳士大学）；University of Dayton（代顿大学）
In this paper, we adopt the maximizing mutual information (MI) approach to tackle the problem of unsupervised learning of binary hash codes for efficient cross-modal retrieval. We proposed a novel method, dubbed Cross-Modal Info-Max Hashing (CMIMH). First, to learn informative representations that can preserve both intra- and inter-modal similarities, we leverage the recent advances in estimating variational lower-bound of MI to maximize the MI between the binary representations and input features and between binary representations of different modalities. By jointly maximizing these MIs under…
Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI
Berlin Institute of Health at Charité - Universitätsmedizin Berlin（柏林健康研究所，柏林夏里特医学院）；Health Data Science Unit, Heidelberg University Hospital and BioQuant（海德堡大学医院与BioQuant健康数据科学部）；Intelligent Medicine Institute, Fudan University（复旦大学智能医学研究所）；Department of Mathematics and Computer Science, Freie Universität Berlin（柏林自由大学数学与计算机科学系）
DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation
State Key Laboratory of Robotics and System, Harbin Institute of Technology（机器人系统国家重点实验室，哈尔滨工业大学）；Institute for Infocomm Research, A*STAR（信息通信研究机构，A*STAR）；College of Information Science and Electronic Engineering, Zhejiang University（信息科学与电子工程学院，浙江大学）
Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two compl…
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Intellifusion Inc., Shenzhen, P.R. China（深圳云天励飞技术股份有限公司）
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor …
University of Electronic Science and Technology of China（电子科技大学）
Existing person re-identification has achieved great progress in the visible domain, capturing all the person images with visible cameras. However, in a 24-hour intelligent surveillance system, the visible cameras may be noneffective at night. In this situation, thermal cameras are the best supplemental components, which capture images without depending on visible light. Therefore, in this paper, we investigate the visible-thermal cross-modality person re-identification (VT Re-ID) problem. In VT Re-ID, there are two knotty problems should be well handled, cross-modality discrepancy and intra-m…
