Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. However, despite showing promising performance, LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We identify two primary reasons for this bias: 1. Different scales of training data between the pretraining stage of LLM and multimodal alignment stage. 2. The learned inference bias due to short-term dependency of text data. Therefore, we propose LACING, a systemic framework designed to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG). Specifically, MDA introduces a parallel dual-attention mechanism that enhances the integration of visual inputs across the model. IFG introduces a learnable soft visual prompt during training and inference to replace visual inputs, designed to compel LVLMs to prioritize text inputs. Then, IFG further proposes a novel decoding strategy using the soft visual prompt to mitigate the model's over-reliance on adjacent text inputs. Comprehensive experiments demonstrate that our method effectively debiases LVLMs from their language bias, enhancing visual comprehension and reducing hallucinations without requiring additional training resources or data. The code and model are available at [lacing-lvlm.github.io](https://lacing-lvlm.github.io).
Renmin University of China（中国人民大学）；Carnegie Mellon University（卡内基梅隆大学）
The topic diversity of open-domain videos leads to various vocabularies and linguistic expressions in describing video contents, and therefore, makes the video captioning task even more challenging. In this paper, we propose an unified caption framework, M&M TGM, which mines multimodal topics in unsupervised fashion from data and guides the caption decoder with these topics. Compared to pre-defined topics, the mined multimodal topics are more semantically and visually coherent and can reflect the topic distribution of videos better. We formulate the topic-aware caption generation as a multi-ta…
XR-CareerAssist: An Immersive Platform for Personalised Career Guidance Leveraging Extended Reality and Multimodal AI
Institute of Communications and Computer Systems (ICCS)（通信与计算机系统研究所）；DASKALOS-APPS；National Technical University of Athens（雅典国家技术大学）；University of Exeter（埃克塞特大学）；CVCOSMOS Ltd（CVCOSMOS有限公司）
INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Communication Content Cognition（通信内容认知技术国家重点实验室）；Peng Cheng Laboratory（鹏城实验室）
Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
Objective, Inc.（奥布杰克蒂夫公司）；Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（鲁德尼大学））；Joint Institute for Nuclear Research（联合核子研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）
University of Oxford（牛津大学）；Tel-Aviv University（特拉维夫大学）
Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly lea…
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what) and mouse traces over an empty canvas (the where) to express the…
Recent years have witnessed astonishing advances in the field of multimodal representation learning, with contrastive learning being the cornerstone for major breakthroughs. Latest works delivered further improvements by incorporating different objectives such as masked modeling and captioning into the frameworks, but our understanding on how these objectives facilitate learning remains vastly incomplete. In this paper, we leverage the fact that classifier-guided diffusion models generate images that reflect the semantic signals provided by the classifier to study the characteristics of multim…
Seoul National University（首尔大学）；AIIS, Seoul National University（首尔大学人工智能研究院）
Critical periods are phases during which a toddler's brain develops in spurts. To promote children's cognitive development, proper guidance is critical in this stage. However, it is not clear whether such a critical period also exists for the training of AI agents. Similar to human toddlers, well-timed guidance and multimodal interactions might significantly enhance the training efficiency of AI agents as well. To validate this hypothesis, we adapt this notion of critical periods to learning in AI agents and investigate the critical period in the virtual environment for AI agents. We formalize…
Image outpainting technology generates visually plausible content regardless of authenticity, making it unreliable to be applied in practice. Thus, we propose a reliable image outpainting task, introducing the sparse depth from LiDARs to extrapolate authentic RGB scenes. The large field view of LiDARs allows it to serve for data enhancement and further multimodal tasks. Concretely, we propose a Depth-Guided Outpainting Network to model different feature representations of two modalities and learn the structure-aware cross-modal fusion. And two components are designed: 1) The Multimodal Learnin…
Digital art synthesis is receiving increasing attention in the multimedia community because of engaging the public with art effectively. Current digital art synthesis methods usually use single-modality inputs as guidance, thereby limiting the expressiveness of the model and the diversity of generated results. To solve this problem, we propose the multimodal guided artwork diffusion (MGAD) model, which is a diffusion-based digital artwork generation approach that utilizes multimodal prompts as guidance to control the classifier-free diffusion model. Additionally, the contrastive language-image…
Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
Harbin Institute of Technology（哈尔滨工业大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济发展实验室）；Chongqing Research Institute of HIT（重庆哈工大研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Tsinghua University（清华大学）；Shenzhen International Graduate School（深圳国际研究生院）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）
Multimodal image registration (MIR) is a fundamental procedure in many image-guided therapies. Recently, unsupervised learning-based methods have demonstrated promising performance over accuracy and efficiency in deformable image registration. However, the estimated deformation fields of the existing methods fully rely on the to-be-registered image pair. It is difficult for the networks to be aware of the mismatched boundaries, resulting in unsatisfactory organ boundary alignment. In this paper, we propose a novel multimodal registration framework, which leverages the deformation fields estima…
Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making
Machine Intelligence, Interaction, and Imagination (Mi3) Laboratory（机器智能、交互与想象实验室）；Laboratory for Intelligent and Safe Automobiles (LISA)（智能与安全汽车实验室）；Johns Hopkins University（约翰霍普金斯大学）；Center for Medicinal Cannabis Research (CMCR)（医药大麻研究中心）
Shanghai Jiao Tong University（上海交通大学）；National Institute of Information（国立信息研究所）
Word representation is a fundamental component in neural language understanding models. Recently, pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally give more accurate contextualized word representations than non-contextualized models do, they are still subject to a sequence of text contexts without diverse hints for word representation from multimodality. This paper thus proposes a visual representation method to explicitly enhance conventional word embeddi…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）；Kuaishou Technology（快手科技）
Multimodal and multi-domain stylization are two important problems in the field of image style transfer. Currently, there are few methods that can perform both multimodal and multi-domain stylization simultaneously. In this paper, we propose a unified framework for multimodal and multi-domain style transfer with the support of both exemplar-based reference and randomly sampled guidance. The key component of our method is a novel style distribution alignment module that eliminates the explicit distribution gaps between various style domains and reduces the risk of mode collapse. The multimodal …
Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error …
EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance
National University of Singapore（新加坡国立大学）；Southern University of Science and Technology（南方科技大学）；Fudan University（复旦大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detection
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute of AI for Industries（工业人工智能研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Medical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice of imaging modality for a particular theranostic task typically involves trade-offs between the feasibility of using a particular modality (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) and the expected performance on a clinical task (e.g., diagnostic accuracy, efficacy of treatment planning and guidance). In this work, we aim to apply the knowledge learned from the less feasible but better-performing (superior) modality to guide the utilization of the more-fe…
In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reco…
Enhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance
School of Computer Science and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机科学与技术学院）；Alibaba Cloud（阿里云）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
Multimodal Image Synthesis and Editing: The Generative AI Era
Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Nanyang Technological University（南洋理工大学）；Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance
School of Artificial Intelligence, Jiangxi Normal University（江西师范大学人工智能学院）；Institute of AI Education, East China Normal University（华东师范大学人工智能教育研究所）；Yale School of Medicine, Yale University（耶鲁大学医学院）
Segment Any Architectural Facades (SAAF):An automatic segmentation model for building facades, walls and windows based on multimodal semantics guidance
National University of Singapore（新加坡国立大学）；Guangzhou University（广州大学）；Tsinghua University（清华大学）；South China University of Technology（华南理工大学）
DeepSPG: Exploring Deep Semantic Prior Guidance for Low-light Image Enhancement with Multimodal Learning
School of Cyber Science and Technology, Hubei University（湖北大学计算机科学与技术学院）；Department of Electrical Automation Design, Beijing Shougang International Engineering Technology（北京首钢国际工程技术部）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学部）；School of Computer Science and Technology, Harbin University of Science and Technology（哈尔滨理工大学计算机科学与技术学院）
FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Physics and Optoelectronic Engineering, Foshan University（佛山大学物理与光电工程学院）；School of Instrumentation Science and Optelectronics Engineering, Beihang University（北航仪器科学与光电工程学院）；School of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）
