M$^{3}$D：面向文档级信息抽取的多模态、多语言、多任务数据集
M$^{3}$D: A Multimodal, Multilingual and Multitask Dataset for Grounded Document-level Information Extraction
AI总结：
针对现有数据集忽视视频多模态和细粒度视觉定位的问题，构建了多模态多语言多任务数据集M$^{3}$D，并提出含去噪特征融合与缺失模态构建模块的分层模型，在英中数据集上取得约53.8%的平均性能。
摘要：
多模态信息抽取（IE）任务日益受到关注，因为许多研究表明多模态信息有助于文本信息抽取。然而，现有的多模态信息抽取数据集主要聚焦于英文文本中基于图像的句子级信息抽取，很少关注基于视频的多模态信息抽取以及细粒度的视觉定位。因此，为了促进多模态信息抽取的发展，我们构建了一个名为M$^{3}$D的多模态、多语言、多任务数据集，其特点如下：（1）包含配对的文档级文本和视频，以丰富多模态信息；（2）支持两种广泛使用的语言，即英语和中文；（3）涵盖更多多模态信息抽取任务，如实体识别、实体链抽取、关系抽取和视觉定位。此外，我们的数据集引入了一个未被探索的主题，即传记，丰富了多模态信息抽取资源的领域。为了为我们的数据集建立基准，我们提出了一种创新的分层多模态信息抽取模型。该模型通过去噪特征融合模块（DFFM）有效利用和整合多模态信息。此外，在非理想场景中，模态信息往往不完整。因此，我们设计了一个缺失模态构建模块（MMCM）来缓解由缺失模态引起的问题。我们的模型在英文和中文数据集的四个任务上分别实现了53.80%和53.77%的平均性能，为后续研究设定了合理的标准。此外，我们进行了更多分析性实验以验证我们提出模块的有效性。我们相信我们的工作能够促进多模态信息抽取领域的发展。
英文摘要：
Multimodal information extraction (IE) tasks have attracted increasing attention because many studies have shown that multimodal information benefits text information extraction. However, existing multimodal IE datasets mainly focus on sentence-level image-facilitated IE in English text, and pay little attention to video-based multimodal IE and fine-grained visual grounding. Therefore, in order to promote the development of multimodal IE, we constructed a multimodal multilingual multitask dataset, named M$^{3}$D, which has the following features: (1) It contains paired document-level text and video to enrich multimodal information; (2) It supports two widely-used languages, namely English and Chinese; (3) It includes more multimodal IE tasks such as entity recognition, entity chain extraction, relation extraction and visual grounding. In addition, our dataset introduces an unexplored theme, i.e., biography, enriching the domains of multimodal IE resources. To establish a benchmark for our dataset, we propose an innovative hierarchical multimodal IE model. This model effectively leverages and integrates multimodal information through a Denoised Feature Fusion Module (DFFM). Furthermore, in non-ideal scenarios, modal information is often incomplete. Thus, we designed a Missing Modality Construction Module (MMCM) to alleviate the issues caused by missing modalities. Our model achieved an average performance of 53.80% and 53.77% on four tasks in English and Chinese datasets, respectively, which set a reasonable standard for subsequent research. In addition, we conducted more analytical experiments to verify the effectiveness of our proposed module. We believe that our work can promote the development of the field of multimodal IE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
