State-of-the-art deep learning algorithms generally require large amounts of data for model training. Lack thereof can severely deteriorate the performance, particularly in scenarios with fine-grained boundaries between categories. To this end, we propose a multimodal approach that facilitates bridging the information gap by means of meaningful joint embeddings. Specifically, we present a benchmark that is multimodal during training (i.e. images and texts) and single-modal in testing time (i.e. images), with the associated task to utilize multimodal data in base classes (with many samples), to learn explicit visual classifiers for novel classes (with few samples). Next, we propose a framework built upon the idea of cross-modal data hallucination. In this regard, we introduce a discriminative text-conditional GAN for sample generation with a simple self-paced strategy for sample selection. We show the results of our proposed discriminative hallucinated method for 1-, 2-, and 5- shot learning on the CUB dataset, where the accuracy is improved by employing multimodal data.
Nanjing University of Science and Technology（南京理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
TIReID aims to retrieve the image corresponding to the given text query from a pool of candidate images. Existing methods employ prior knowledge from single-modality pre-training to facilitate learning, but lack multi-modal correspondences. Besides, due to the substantial gap between modalities, existing methods embed the original modal features into the same latent space for cross-modal alignment. However, feature embedding may lead to intra-modal information distortion. Recently, CLIP has attracted extensive attention from researchers due to its powerful semantic concept learning capacity an…
Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Innovation Center of Calligraphy and Painting Creation Technology, MCT（中国传媒书画创作技术创新中心）；Videt Technology（维德科技）
Scene text instances found in natural images carry explicit semantic information that can provide important cues to solve a wide array of computer vision problems. In this paper, we focus on leveraging multi-modal content in the form of visual and textual cues to tackle the task of fine-grained image classification and retrieval. First, we obtain the text instances from images by employing a text reading system. Then, we combine textual features with salient image regions to exploit the complementary information carried by the two sources. Specifically, we employ a Graph Convolutional Network …
University of Exeter（埃克塞特大学）；University of Tübingen（图宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Surrey（萨里大学）
Representation learning for sketch-based image retrieval has mostly been tackled by learning embeddings that discard modality-specific information. As instances from different modalities can often provide complementary information describing the underlying concept, we propose a cross-attention framework for Vision Transformers (XModalViT) that fuses modality-specific information instead of discarding them. Our framework first maps paired datapoints from the individual photo and sketch modalities to fused representations that unify information from both modalities. We then decouple the input sp…
University of Cambridge（剑桥大学）；Huawei R&D UK（华为英国研发中心）
Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and a large amount of computation resources due to the noise present in the uncurated data. This is partly due to the fact that the prevalent training scheme is trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated wit…
Arizona State University（亚利桑那州立大学）；Duke University（杜克大学）
Real-world robotics systems deal with data from a multitude of modalities, especially for tasks such as navigation and recognition. The performance of those systems can drastically degrade when one or more modalities become inaccessible, due to factors such as sensors' malfunctions or adverse environments. Here, we argue modality hallucination as one effective way to ensure consistent modality availability and thereby reduce unfavorable consequences. While hallucinating data from a modality with richer information, e.g., RGB to depth, has been researched extensively, we investigate the more ch…
SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所人工智能安全重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Trustworthy Technology and Engineering Laboratory, Huawei（华为可信技术与工程实验室）
Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Meituan（美团）；Zhejiang-Singapore Innovation and AI Joint Research Lab, Zhejiang University（浙江大学浙新创新与AI联合研究实验室）
DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
Tsinghua University, Tencent（清华大学、腾讯）；Tencent（腾讯）；Tencent, University of Macau（腾讯、澳门大学）；University of Science and Technology Beijing（北京科技大学）；Tsinghua University（清华大学）
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）；Meituan（美团）；Inspur Database Technology（Inspur数据库技术）；Guilin University of Electronic Technology（桂林电子科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
Hanoi University of Science and Technology（河内科技大学）；Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Nanjing University（南京大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Hanoi-Amsterdam High School for the Gifted（河内阿姆斯特丹天才高中）
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
University of Helsinki（赫尔辛基大学）；Politecnico di Torino（都灵理工大学）；Universiteit Utrecht（乌得勒支大学）；Université Bretagne Sud（南布列塔尼大学）；University of Copenhagen（哥本哈根大学）；University Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；University of Lorraine（洛林大学）
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Tsinghua University（清华大学）；Tsinghua University, Tencent（清华大学腾讯）；Tencent（腾讯）；University of Science and Technology Beijing（北京科技大学）；University of Macau（澳门大学）
Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing
School of Computer Science, Wuhan University, China（武汉大学计算机学院）；Zhongguancun Academy, China（中关村学院）；School of Computer Science, Chongqing University, China（重庆大学计算机学院）
Learning Fine-grained Domain Generalization via Hyperbolic State Space Hallucination
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Faculty of Information Technology, Monash University（莫纳什大学信息技术学院）；School of Medicine, Yale University（耶鲁大学医学院）
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
The University of Manchester（曼彻斯特大学）；The University of Hong Kong（香港大学）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Academy of Educational Sciences（上海市教育科学研究院）；National University of Singapore（新加坡国立大学）
Thapar Institute of Engineering and Technology（塔帕尔工程技术学院）；Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）；Oxford Brookes University（牛津布鲁克斯大学）
Recently, there has been numerous breakthroughs in face hallucination tasks. However, the task remains rather challenging in videos in comparison to the images due to inherent consistency issues. The presence of extra temporal dimension in video face hallucination makes it non-trivial to learn the facial motion through out the sequence. In order to learn these fine spatio-temporal motion details, we propose a novel cross-modal audio-visual Video Face Hallucination Generative Adversarial Network (VFH-GAN). The architecture exploits the semantic correlation of between the movement of the facial …
Robust Multimodal Large Language Models Against Modality Conflict
School of Artificial Intelligence（人工智能学院）；Data Science, University of Science（数据科学，科学大学）；Department of Electronic Engineering（电子工程系）；Information Science, University of Science（信息科学，科学大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）
We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI). We address two major shortcomings of standard multimodal approaches, limited area coverage and reduced reliability. Our new framework fuses the concept of modality hallucination with triplet learning to train a model with different modalities to handle missing sensors at inference time. We evaluate the proposed model on inertial data from a wearable accelerometer device, using RGB videos and skeletons as privileged modalities, and show…
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Helmholtz Munich（海德堡-慕尼黑亥姆霍尔茨中心）；Google（谷歌）；LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎电信学院，巴黎理工学院）
We deal with the problem of information fusion driven satellite image/scene classification and propose a generic hallucination architecture considering that all the available sensor information are present during training while some of the image modalities may be absent while testing. It is well-known that different sensors are capable of capturing complementary information for a given geographical area and a classification module incorporating information from all the sources are expected to produce an improved performance as compared to considering only a subset of the modalities. However, t…
