UMIE：基于指令微调的统一多模态信息抽取
UMIE: Unified Multimodal Information Extraction with Instruction Tuning
AI总结：
本文提出统一的多模态信息抽取器UMIE，通过指令微调将三个MIE任务统一为生成问题，有效抽取文本和视觉提及，在多个数据集上超越现有方法并具备强泛化能力。
摘要：
随着多媒体内容的普及，多模态信息抽取（MIE）备受关注。然而，当前的MIE方法通常采用任务特定的模型结构，导致跨任务的泛化能力受限，且未能充分利用MIE任务间的共享知识。为解决这些问题，我们提出了UMIE，一个统一的多模态信息抽取器，通过指令微调将三个MIE任务统一为一个生成问题，能够有效抽取文本和视觉提及。大量实验表明，我们单一的UMIE在三个任务的六个MIE数据集上优于各种最先进的（SoTA）方法。此外，深入分析证明了UMIE在零样本设置下的强泛化能力、对指令变体的鲁棒性以及可解释性。我们的研究是迈向统一MIE模型的初步步骤，并开启了对MIE领域内指令微调和大语言模型的探索。我们的代码、数据和模型可在 https://github.com/ZUCC-AI/UMIE 获取。
英文摘要：
Multimodal information extraction (MIE) gains significant attention as the popularity of multimedia content increases. However, current MIE methods often resort to using task-specific model structures, which results in limited generalizability across tasks and underutilizes shared knowledge across MIE tasks. To address these issues, we propose UMIE, a unified multimodal information extractor to unify three MIE tasks as a generation problem using instruction tuning, being able to effectively extract both textual and visual mentions. Extensive experiments show that our single UMIE outperforms various state-of-the-art (SoTA) methods across six MIE datasets on three tasks. Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the zero-shot setting, robustness to instruction variants, and interpretability. Our research serves as an initial step towards a unified MIE model and initiates the exploration into both instruction tuning and large language models within the MIE domain. Our code, data, and model are available at https://github.com/ZUCC-AI/UMIE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
