Vision Language Models (VLMs) are typically evaluated with Visual Question Answering (VQA) tasks which assess a model's understanding of scenes. Good VQA performance is taken as evidence that the model will perform well on a broader range of tasks that require both visual and language inputs. However, scene-aware VQA does not fully capture input biases or assess hallucinations caused by a misalignment between modalities. To address this, we propose a Cross-modal Alignment Similarity Test (CAST) to probe VLMs for self-consistency across modalities. This test involves asking the models to identify similarities between two scenes through text-only, image-only, or both and then assess the truthfulness of the similarities they generate. Since there is no ground-truth to compare against, this evaluation does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. We argue that while not all self-consistent models are capable or accurate, all capable VLMs must be self-consistent.
Variational Adapter for Cross-modal Similarity Representation
School of Remote Sensing and Information Engineering(遥感与信息工程学院);Wuhan University(武汉大学);School of Data Science and Engineering(数据科学与工程学院);East China Normal University(华东师范大学);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing(测绘遥感信息工程国家重点实验室)
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Intellifusion Inc., Shenzhen, P.R. China(深圳云天励飞技术股份有限公司)
有效检测假新闻最近引起了极大的关注。目前的研究在预测假新闻方面做出了重大贡献,但较少关注利用新闻文章中文本和视觉信息之间的关系(相似性)。重视这种相似性有助于识别假新闻故事,例如,试图使用不相关的图像来吸引读者的注意力。在这项工作中,我们提出了一个$\mathsf{S}$ imilarity- $\mathsf{A}$ ware $\mathsf{F}$ ak $\mathsf{E}$新闻检测方法($\mathsf{SAFE}$),调查多模态(文本和视觉)信息的新闻文章。首先,采用神经网络分别提取新闻文本和视觉特征。我们进一步研究跨模态提取的特征之间的关系。新闻文本和视觉信息的这种表示及其关系被联合学习并用于预测假新闻。提出的方法有助于识别虚假的新闻文章的基础上,他们的文字,图像,或他们的“不匹配”。“我们对大规模的真实数据进行了广泛的实验,证明了所提出的方法的有效性。
M$^2$S-Net: Multi-Modal Similarity Metric Learning based Deep Convolutional Network for Answer Selection
Shenma Search, Alibaba Mobile GroupAlibaba-Inc(阿里巴巴移动事业群神马搜索阿里巴巴公司);Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)Institute of Computing Technology, CAS(中国科学院智能信息处理重点实验室中国科学院计算技术研究所)
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Institute of Information Engineering, Chinese Academy of Sciences(信息工程研究所,中国科学院);School of Cyber Security, University of Chinese Academy of Sciences(网络安全学院,中国科学院大学)