Attention models are widely used in Vision-language (V-L) tasks to perform the visual-textual correlation. Humans perform such a correlation with a strong linguistic understanding of the visual world. However, even the best performing attention model in V-L tasks lacks such a high-level linguistic understanding, thus creating a semantic gap between the modalities. In this paper, we propose an attention mechanism - Linguistically-aware Attention (LAT) - that leverages object attributes obtained from generic object detectors along with pre-trained language models to reduce this semantic gap. LAT represents visual and textual modalities in a common linguistically-rich space, thus providing linguistic awareness to the attention process. We apply and demonstrate the effectiveness of LAT in three V-L tasks: Counting-VQA, VQA, and Image captioning. In Counting-VQA, we propose a novel counting-specific VQA model to predict an intuitive count and achieve state-of-the-art results on five datasets. In VQA and Captioning, we show the generic nature and effectiveness of LAT by adapting it into various baselines and consistently improving their performance.
Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models
School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；Zhongguancun Academy（中关村学院）；Southeast Academy of Information Technology, Beijing Institute of Technology（信息技术东南学院，北京理工大学）；Tsinghua University（清华大学）
MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting
Department of Biomedical Engineering（生物医学工程系）；Georgia Institute of Technology（佐治亚理工学院）；Department of Machine Learning（机器学习系）；Department of Radiation Oncology（放射肿瘤科）；Emory University School of Medicine（埃默里大学医学院）；University of Southern California（南加州大学）
HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
Netaji Subhash Engineering College (NSEC)（奈尔贾伊·萨布哈工程学院）；TCG Crest；Electronics and Communication Sciences Unit (ECSU)（电子与通信科学单位）；Indian Statistical Institute（印度统计研究所）
Reducing Annotation Burden in Physical Activity Research Using Vision-Language Models
Department of Population Health, University of Oxford（牛津大学流行病学与健康统计学系）；University of Oxford（牛津大学）；School of Epidemiology and Health Statistics, Chengdu Medical College（成都医学院流行病学与健康统计学系）
Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, University of Wisconsin–Madison（威斯康星大学麦迪逊分校计算机科学系）；School of Engineering Science, University of the Chinese Academy of Sciences（中国科学院工程科学学院）；NExT++ Research Centre, National University of Singapore（新加坡国立大学NExT++研究中心）
VLMs Guided Interpretable Decision Making for Autonomous Driving
Department of Computer Science, Tulane University（路易斯安那大学计算机科学系）；Qualcomm（高通公司）；Department of Industrial and Systems Engineering, North Carolina State University（北卡罗来纳州立大学工业与系统工程系）
Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. The occurrences of coreference relations in the dialog makes it a more challenging task than visual question-answering. Most previous works have focused on learning better multi-modal representations or on exploring different ways of fusing visual and language features, while the coreferences in the dialog are mainly ignored. In this paper, based on linguistic knowledge and discourse features of human dialog we propose…
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Kuaishou Technology（快手科技）；School of Software Technology, Zhejiang University（浙江大学软件学院）
Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence（计算机视觉系，Mohamed Bin Zayed人工智能大学）；School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学）；Bytedance Seed（字节跳动种子）；School of Computer Science, Qinghai Normal University（计算机科学学院，青海师范大学）
CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
School of Information Engineering, Minzu University of China（中国民族大学信息工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；National Library of China, Beijing, China（中国国家图书馆）；School of Cyberspace Security, Beijing University of Posts and Telecommunications（北京邮电大学网络安全学院）
UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；NTU（南洋理工大学）；University of Chicago（芝加哥大学）；NUS（新加坡国立大学）；University of Maryland（马里兰大学）；Rutgers University（罗格斯大学）；HKUST（香港科技大学）；PolyU（香港理工大学）
We present an empirical analysis of the state-of-the-art systems for referring expression recognition -- the task of identifying the object in an image referred to by a natural language expression -- with the goal of gaining insight into how these systems reason about language and vision. Surprisingly, we find strong evidence that even sophisticated and linguistically-motivated models for this task may ignore the linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. For example, we show that a system trained …
DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness
Department of Computer Science（计算机科学系）；Engineering, Ohio State University, Ohio, US（工程系，俄亥俄州立大学，俄亥俄，美国）；Department of Computer Science, University of Central Florida, Florida, US（计算机科学系，中央佛罗里达大学，佛罗里达，美国）
VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculture
Department of Electrical and Computer Engineering, The University of Alabama（电气与计算机工程系，阿拉巴马大学）；Electrical and Computer Engineering, Michigan State University（电气与计算机工程，密歇根州立大学）；Agricultural and Biological Engineering, Mississippi State University（农业与生物工程，密苏里州立大学）；Department of Computer Science, University of Alabama（计算机科学系，阿拉巴马大学）；USDA-ARS Genetics and Sustainbale Agriculture（美国农业部ARS基因与可持续农业）
Beijing Normal University（北京师范大学）；University of Waterloo（滑铁卢大学）；EPFL（洛桑联邦理工学院）
Standard 3D convolution operations require much larger amounts of memory and computation cost than 2D convolution operations. The fact has hindered the development of deep neural nets in many 3D vision tasks. In this paper, we investigate the possibility of applying depthwise separable convolutions in 3D scenario and introduce the use of 3D depthwise convolution. A 3D depthwise convolution splits a single standard 3D convolution into two separate steps, which would drastically reduce the number of parameters in 3D convolutions with more than one order of magnitude. We experiment with 3D depthw…
