Large Multimodal Models (LMMs) are powerful tools that are capable of reasoning and understanding multimodal information beyond text and language. Despite their entrenched impact, the development of LMMs is hindered by the higher computational requirements compared to their unimodal counterparts. One of the main causes of this is the large amount of tokens needed to encode the visual input, which is especially evident for multi-image multimodal tasks. Recent approaches to reduce visual tokens depend on the visual encoder architecture, require fine-tuning the LLM to maintain the performance, and only consider single-image scenarios. To address these limitations, we propose ToFu, a visual encoder-agnostic, training-free Token Fusion strategy that combines redundant visual tokens of LMMs for high-resolution, multi-image, tasks. The core intuition behind our method is straightforward yet effective: preserve distinctive tokens while combining similar ones. We achieve this by sequentially examining visual tokens and deciding whether to merge them with others or keep them as separate entities. We validate our approach on the well-established LLaVA-Interleave Bench, which covers challenging multi-image tasks. In addition, we push to the extreme our method by testing it on a newly-created benchmark, ComPairs, focused on multi-image comparisons where a larger amount of images and visual tokens are inputted to the LMMs. Our extensive analysis, considering several LMM architectures, demonstrates the benefits of our approach both in terms of efficiency and performance gain.
A micro-expression is a spontaneous unconscious facial muscle movement that can reveal the true emotions people attempt to hide. Although manual methods have made good progress and deep learning is gaining prominence. Due to the short duration of micro-expression and different scales of expressed in facial regions, existing algorithms cannot extract multi-modal multi-scale facial region features while taking into account contextual information to learn underlying features. Therefore, in order to solve the above problems, a multi-modal multi-scale algorithm based on transformer network is propo…
Normandie Univ（诺曼底大学）；UNICAEN（卡昂大学）；ENSICAEN（卡昂国立高等工程师学院）；CNRS -- UMR GREYC（法国国家科学研究中心——GREYC联合研究单位）
Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra c…
Yonsei University（延世大学）；Chungnam National University（忠南国立大学）；INRIA（法国国家信息与自动化研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Establishing dense correspondences between multiple images is a fundamental task in many applications. However, finding a reliable correspondence in multi-modal or multi-spectral images still remains unsolved due to their challenging photometric and geometric variations. In this paper, we propose a novel dense descriptor, called dense adaptive self-correlation (DASC), to estimate multi-modal and multi-spectral dense correspondences. Based on an observation that self-similarity existing within images is robust to imaging modality variations, we define the descriptor with a series of an adaptive…
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore（新加坡国立大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；National Neuroscience Institute（国家神经科学研究所）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）
Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
Nagoya University（名古屋大学）；Northeastern University（东北大学）；KC Machine Learning Lab（KC机器学习实验室）；University of Science, Vietnam National University Ho Chi Minh City（胡志明市越南国立大学理科大学）；University of California, Los Angeles（加州大学洛杉矶分校）
A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Jinfeng Laboratory（金凤实验室）；Shandong First Medical University（山东第一医科大学）；Shandong Provincial Qianfoshan Hospital（山东省千佛山医院）；The Third Affiliated Hospital of Kunming Medical University（昆明医科大学第三附属医院）；Yunnan Cancer Hospital（云南省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）；Prince of Wales Hospital（威尔士亲王医院）；Sun Yat-sen University（中山大学）
Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media
L3S Research Center \ University Hannover Hannover Germany；Indian Institute of Technology Kharagpur Kharagpur India；L3S Research Center\ University Hannover Hannover Germany；L3S Research Center \ University Hannover；Indian Institute of Technology Kharagpur；L3S Research Center\ University Hannover
Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a…
Contrastive learning-based vision-language pre-training approaches, such as CLIP, have demonstrated great success in many vision-language tasks. These methods achieve cross-modal alignment by encoding a matched image-text pair with similar feature embeddings, which are generated by aggregating information from visual patches and language tokens. However, direct aligning cross-modal information using such representations is challenging, as visual patches and text tokens differ in semantic levels and granularities. To alleviate this issue, we propose a Finite Discrete Tokens (FDT) based multimod…
Recently, contrastive learning-based image translation methods have been proposed, which contrasts different spatial locations to enhance the spatial correspondence. However, the methods often ignore the diverse semantic relation within the images. To address this, here we propose a novel semantic relation consistency (SRC) regularization along with the decoupled contrastive learning, which utilize the diverse semantics by focusing on the heterogeneous semantics between the image patches of a single image. To further improve the performance, we present a hard negative mining by exploiting the …
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Technology Sydney（悉尼科技大学）；University of Oxford（牛津大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；DarkMatter AI Research（DarkMatter AI 研究院）
Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
South China University of Technology（华南理工大学）；Institute for Infocomm Research (I 2 R), A*STAR（信息通信研究所（I 2 R），A*STAR）；WeChat Vision, Tencent Inc.（微信视觉，腾讯公司）；Foshan University（佛山大学）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
Various multi-modal imaging sensors are currently involved at different steps of an interventional therapeutic work-flow. Cone beam computed tomography (CBCT), computed tomography (CT) or Magnetic Resonance (MR) images thereby provides complementary functional and/or structural information of the targeted region and organs at risk. Merging this information relies on a correct spatial alignment of the observed anatomy between the acquired images. This can be achieved by the means of multi-modal deformable image registration (DIR), demonstrated to be capable of estimating dense and elastic defor…
We present a novel descriptor, called deep self-convolutional activations (DeSCA), designed for establishing dense correspondences between images taken under different imaging modalities, such as different spectral ranges or lighting conditions. Motivated by descriptors based on local self-similarity (LSS), we formulate a novel descriptor by leveraging LSS in a deep architecture, leading to better discriminative power and greater robustness to non-rigid image deformations than state-of-the-art cross-modality descriptors. The DeSCA first computes self-convolutions over a local support window fo…
Fuzhou University（福州大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）
Both image registration and label fusion in the multi-atlas segmentation (MAS) rely on the intensity similarity between target and atlas images. However, such similarity can be problematic when target and atlas images are acquired using different imaging protocols. High-level structure information can provide reliable similarity measurement for cross-modality images when cooperating with deep neural networks (DNNs). This work presents a new MAS framework for cross-modality images, where both image registration and label fusion are achieved by DNNs. For image registration, we propose a consiste…
Nanjing University of Science and Technology（南京理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. Existing methods employ prior knowledge from single-modality pre-training to facilitate learning, but lack multi-modal correspondences. Besides, due to the substantial gap between modalities, existing methods embed the original modal features into the same latent space for cross-modal alignment. However, feature embedding may lead to intra-modal information distortion. Recently, CLIP has attracted extensive attention from researchers due to its powerful semantic concept learning capacity an…
The security of the Person Re-identification(ReID) model plays a decisive role in the application of ReID. However, deep neural networks have been shown to be vulnerable, and adding undetectable adversarial perturbations to clean images can trick deep neural networks that perform well in clean images. We propose a ReID multi-modal data augmentation method with adversarial defense effect: 1) Grayscale Patch Replacement, it consists of Local Grayscale Patch Replacement(LGPR) and Global Grayscale Patch Replacement(GGPR). This method can not only improve the accuracy of the model, but also help th…
Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification
Research Center for Medical Image Analysis and Artificial Intelligence（医学图像分析与人工智能研究中心）；Department of Medicine（医学系）；Faculty of Medicine and Dentistry（医学与牙科学院）；Danube Private University（多瑙私立大学）
University of Michigan（密歇根大学）；Royal Observatory of Belgium（比利时皇家天文台）；National Solar Observatory（美国国家太阳天文台）
Sunspots, as seen in white light or continuum images, are associated with regions of high magnetic activity on the Sun, visible on magnetogram images. Their complexity is correlated with explosive solar activity and so classifying these active regions is useful for predicting future solar activity. Current classification of sunspot groups is visually based and suffers from bias. Supervised learning methods can reduce human bias but fail to optimally capitalize on the information present in sunspot images. This paper uses two image modalities (continuum and magnetogram) to characterize the spat…
2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds
National University of Defense Technology（国防科技大学）；Defense Innovation Institute, Academy of Military Sciences（军事科学院国防科技创新研究院）；Sun Yat-sen University（中山大学）
Document structure extraction has been a widely researched area for decades. Recent work in this direction has been deep learning-based, mostly focusing on extracting structure using fully convolution NN through semantic segmentation. In this work, we present a novel multi-modal approach for form structure extraction. Given simple elements such as textruns and widgets, we extract higher-order structures such as TextBlocks, Text Fields, Choice Fields, and Choice Groups, which are essential for information collection in forms. To achieve this, we obtain a local image patch around each low-level …
RemoteDet-Mamba: A Hybrid Mamba-CNN Network for Multi-modal Object Detection in Remote Sensing Images
School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China（计算机科学学院，北京邮电大学）；School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China（电子工程学院，北京邮电大学）
University of Kansas（堪萨斯大学）；NUST（巴基斯坦国立科技大学）；Ryerson University（瑞尔森大学）
Crowd estimation is a very challenging problem. The most recent study tries to exploit auditory information to aid the visual models, however, the performance is limited due to the lack of an effective approach for feature extraction and integration. The paper proposes a new audiovisual multi-task network to address the critical challenges in crowd counting by effectively utilizing both visual and audio inputs for better modalities association and productive feature extraction. The proposed network introduces the notion of auxiliary and explicit image patch-importance ranking (PIR) and patch-w…
On-Device Inference versus Wireless Streaming: Energy-Efficient Multi-Modal Deep Learning for Wearable Cardiovascular Patches
CTO System Innovation, NXP Semiconductors Germany GmbH（NXP半导体德国系统创新部）；Advanced Chip Engineering, NXP Semiconductors（NXP半导体先进芯片工程部）；Business Line Secure Connected Edge, NXP Semiconductors（NXP半导体安全连接边缘业务线）；Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology（汉堡技术大学医学技术与智能系统研究所）；Department of Informatics, Hamburg University of Applied Sciences（汉堡应用科学大学信息学院）
Self-supervised learning (SSL) enables learning useful inductive biases through utilizing pretext tasks that require no labels. The unlabeled nature of SSL makes it especially important for whole slide histopathological images (WSIs), where patch-level human annotation is difficult. Masked Autoencoders (MAE) is a recent SSL method suitable for digital pathology as it does not require negative sampling and requires little to no data augmentations. However, the domain shift between natural images and digital pathology images requires further research in designing MAE for patch-level WSIs. In thi…
School of Computer Science, Southeast University（东南大学计算机科学与工程学院）
Most of existing RGB-D salient object detection (SOD) methods follow the CNN-based paradigm, which is unable to model long-range dependencies across space and modalities due to the natural locality of CNNs. Here we propose the Hierarchical Cross-modal Transformer (HCT), a new multi-modal transformer, to tackle this problem. Unlike previous multi-modal transformers that directly connecting all patches from two modalities, we explore the cross-modal complementarity hierarchically to respect the modality gap and spatial discrepancy in unaligned regions. Specifically, we propose to use intra-modal…
SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）智能制造学域）；College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与运载工程学院）；The Hong Kong University of Science and Technology（香港科技大学）
Technische Universität Berlin（柏林工业大学）；Direção-Geral do Território (DGT)（领土总局）；NOVA Information Management School (NOVA IMS), Universidade Nova Lisboa（里斯本新大学NOVA信息管理学院）
This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) B…
Recent studies suggest that combined analysis of Magnetic resonance imaging~(MRI) that measures brain atrophy and positron emission tomography~(PET) that quantifies hypo-metabolism provides improved accuracy in diagnosing Alzheimer's disease. However, such techniques are limited by the availability of corresponding scans of each modality. Current work focuses on a cross-modal approach to estimate FDG-PET scans for the given MR scans using a 3D U-Net architecture. The use of the complete MR image instead of a local patch based approach helps in capturing non-local and non-linear correlations be…
The University of Sheffield（谢菲尔德大学）；University of East Anglia（东英吉利大学）
Cross-modal image synthesis is a topical problem in medical image computing. Existing methods for image synthesis are either tailored to a specific application, require large scale training sets, or are based on partitioning images into overlapping patches. In this paper, we propose a novel Dual cOnvolutional filTer lEarning (DOTE) approach to overcome the drawbacks of these approaches. We construct a closed loop joint filter learning strategy that generates informative feedback for model self-optimization. Our method can leverage data more efficiently thus reducing the size of the required tr…
