arXiv 2402.099892024-05-29cs.CVcs.CL

LLMs作为桥梁：重构基于多模态命名实体识别

LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition

School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Mathematics, Taiyuan University of Technology（太原理工大学数学学院）；Tianjin University of Science and Technology（天津科技大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；University of Copenhagen（哥本哈根大学）

Jinyuan Li, Han Li, Di Sun, Jiahao Wang, Wenkun Zhang, Zan Wang, Gang Pan

备注：: Accepted to Findings of ACL 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出RiVEG框架，利用大型语言模型作为桥梁，将基于多模态命名实体识别重构为联合任务，统一视觉接地与实体接地，在三个子任务上分别取得10.65%、6.21%和8.83%的绝对领先。

摘要：

基于多模态命名实体识别（GMNER）是一项新兴的多模态任务，旨在识别命名实体、实体类型及其对应的视觉区域。GMNER任务展现出两个具有挑战性的特性：1）社交媒体中图像-文本对的弱相关性导致相当一部分命名实体无法被接地（即无法定位到视觉区域）。2）在类似任务（如短语定位、指代表达式理解）中常用的粗粒度指代表达式与细粒度命名实体之间存在区别。在本文中，我们提出RiVEG，一个统一框架，通过利用大型语言模型（LLMs）作为连接桥梁，将GMNER重构为联合的MNER-VE-VG任务。这种重构带来两个好处：1）它保持了最优的MNER性能，并消除了使用目标检测方法预提取区域特征的需求，从而自然地解决了现有GMNER方法的两个主要局限性。2）引入实体扩展表达式和视觉蕴含（VE）模块，统一了视觉接地（VG）和实体接地（EG）。这使得RiVEG能够轻松继承当前或未来任何多模态预训练模型的视觉蕴含和视觉接地能力。大量实验表明，在现有GMNER数据集上，RiVEG优于最先进的方法，并在所有三个子任务中分别取得了10.65%、6.21%和8.83%的绝对领先优势。

英文摘要：

Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging properties: 1) The weak correlation between image-text pairs in social media results in a significant portion of named entities being ungroundable. 2) There exists a distinction between coarse-grained referring expressions commonly used in similar tasks (e.g., phrase localization, referring expression comprehension) and fine-grained named entities. In this paper, we propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as a connecting bridge. This reformulation brings two benefits: 1) It maintains the optimal MNER performance and eliminates the need for employing object detection methods to pre-extract regional features, thereby naturally addressing two major limitations of existing GMNER methods. 2) The introduction of entity expansion expression and Visual Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding capabilities of any current or prospective multimodal pretraining models. Extensive experiments demonstrate that RiVEG outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。