Conditional inference on joint textual and visual clues is a multi-modal reasoning task that textual clues provide prior permutation or external knowledge, which are complementary with visual content and pivotal to deducing the correct option. Previous methods utilizing pretrained vision-language models (VLMs) have achieved impressive performances, yet they show a lack of multimodal context reasoning capability, especially for text-modal information. To address this issue, we propose a Multi-modal Context Reasoning approach, named ModCR. Compared to VLMs performing reasoning via cross modal semantic alignment, it regards the given textual abstract semantic and objective image information as the pre-context information and embeds them into the language model to perform context reasoning. Different from recent vision-aided language models used in natural language processing, ModCR incorporates the multi-view semantic alignment information between language and vision by introducing the learnable alignment prefix between image and text in the pretrained language model. This makes the language model well-suitable for such multi-modal reasoning scenario on joint textual and visual clues. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance (exact gain by 4.8% on PMR test set) compared to previous strong baselines. Code Link: \url{https://github.com/YunxinLi/Multimodal-Context-Reasoning}.
Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencing
School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand（梅西大学数学与计算科学学院）；School of Computer Science and Informatics, Cardiff University, Cardiff, UK（卡迪夫大学计算机科学与信息学院）
Using textual clues to improve metaphor processing
在本文中，我们提出了一个文本线索的方法来帮助隐喻检测，以提高语义处理的这一数字。以往的研究都是只研究语义上的语义关系，而忽略了一组明显的语义关系。基于语料库的分析表明，隐喻的表层隐喻是存在的。这些线索可以通过句法结构和词汇标记来表征。我们提出了一个面向对象的模型来表示文本的线索，发现。这种表示的目的是帮助选择的语义处理，在可能的非字面意义。目前正在开发一个实施这一模式的原型，采用一种渐进的方法，以便逐步进行评价。\footnote{This work takes part in a research project sponsored by the AUPELF-UREF (Francophone Agency For Education and Research)}
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition
Shopee（虾皮）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉成像工程技术研究中心）；Microsoft（微软）；Alibaba Group（阿里巴巴集团）
Multi-Granular Multimodal Clue Fusion for Meme Understanding
School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）；National University of Singapore（新加坡国立大学）；North China Institute of Computing Technology（华北计算技术研究所）
MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval
University of Sheffield（谢菲尔德大学）；University of Science and Technology Beijing（北京科技大学）；University of California San Diego（加州大学圣地亚哥分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval
State Key Laboratory of General Artificial Intelligence, Peking University（北京理工大学通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；Key Laboratory of Target Cognition and Application Technology（目标认知与应用技术重点实验室）；Beijing Institute of Technology（北京理工大学）；Ucap Cloud（Ucap云）
Named Entity and Relation Extraction with Multi-Modal Retrieval
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Singapore University of Technology and Design（新加坡科技设计大学）
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Digital China Group（数字中国集团）
CLEAR: Character Unlearning in Textual and Visual Modalities
AIRI；HSE University（俄罗斯高等经济大学）；Skoltech（斯克里普契克技术大学）；Sber AI（俄罗斯储蓄银行人工智能实验室）；MTUCI（莫斯科国立大学）；MIPT（米哈伊尔戈尔斯基理工学院）；ISP RAS Research Center for Trusted AI（俄罗斯科学院信息与系统问题研究所可信人工智能研究中心）
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
MICS, CentraleSupélec - Université Paris-Saclay, France（MICS，中央圣艾尔布兰大学-巴黎萨克雷大学，法国）；Google DeepMind, London, UK（谷歌DeepMind，伦敦，英国）；CONICET, Universidad de Buenos Aires, Argentina（CONICET，布宜诺斯艾利斯大学，阿根廷）
Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Mashang Consumer Finance Co., Ltd.（马上消费金融股份有限公司）
有效检测假新闻最近引起了极大的关注。目前的研究在预测假新闻方面做出了重大贡献，但较少关注利用新闻文章中文本和视觉信息之间的关系（相似性）。重视这种相似性有助于识别假新闻故事，例如，试图使用不相关的图像来吸引读者的注意力。在这项工作中，我们提出了一个$\mathsf{S}$ imilarity- $\mathsf{A}$ ware $\mathsf{F}$ ak $\mathsf{E}$新闻检测方法（$\mathsf{SAFE}$），调查多模态（文本和视觉）信息的新闻文章。首先，采用神经网络分别提取新闻文本和视觉特征。我们进一步研究跨模态提取的特征之间的关系。新闻文本和视觉信息的这种表示及其关系被联合学习并用于预测假新闻。提出的方法有助于识别虚假的新闻文章的基础上，他们的文字，图像，或他们的“不匹配”。“我们对大规模的真实数据进行了广泛的实验，证明了所提出的方法的有效性。
