多模态关系抽取任务已引起研究界的广泛关注，但其进展受到可用训练数据稀缺的限制。一个自然的想法是利用跨模态生成模型扩展现有数据集。本文考虑一种新颖的问题设定：训练期间仅有单模态数据（文本或图像）可用。我们的目标是利用合成数据训练一个多模态分类器，使其在真实多模态测试数据上表现良好。然而，使用合成数据训练面临两个障碍：数据多样性不足以及标签信息丢失。为缓解这些问题，我们提出了互信息感知的多模态迭代关系数据生成方法 MI2RAGE（Mutual Information-aware Multimodal Iterated Relational dAta GEneration）。该方法采用链式跨模态生成（Chained Cross-modal Generation，CCG）来提升生成数据的多样性，并利用教师网络选择与真实标签具有高互信息的有价值训练样本。将我们的方法与直接在合成数据上训练相比，使用合成文本时 F1 显著提升 24.06%，使用合成图像时 F1 显著提升 26.42%。值得注意的是，我们完全在合成图像上训练的最佳模型，在 F1 上比此前在真实多模态数据上训练的最先进模型高出 3.76%。代码库将在论文被接收后公开。
英文摘要：
The task of multimodal relation extraction has attracted significant research attention, but progress is constrained by the scarcity of available training data. One natural thought is to extend existing datasets with cross-modal generative models. In this paper, we consider a novel problem setting, where only unimodal data, either text or image, are available during training. We aim to train a multimodal classifier from synthetic data that perform well on real multimodal test data. However, training with synthetic data suffers from two obstacles: lack of data diversity and label information loss. To alleviate the issues, we propose Mutual Information-aware Multimodal Iterated Relational dAta GEneration (MI2RAGE), which applies Chained Cross-modal Generation (CCG) to promote diversity in the generated data and exploits a teacher network to select valuable training samples with high mutual information with the ground-truth labels. Comparing our method to direct training on synthetic data, we observed a significant improvement of 24.06% F1 with synthetic text and 26.42% F1 with synthetic images. Notably, our best model trained on completely synthetic images outperforms prior state-of-the-art models trained on real multimodal data by a margin of 3.76% in F1. Our codebase will be made available upon acceptance.
Multimodal Learning on Graphs for Disease Relation Extraction
Beijing Institute of Technology（北京理工大学）；University of Southern California（南加州大学）；Tsinghua University（清华大学）；Harvard T.H.Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Harvard Medical School（哈佛医学院）；Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学博德研究所）；Harvard Data Science Initiative（哈佛数据科学研究院）
M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction
Yokohama National University（Yokohama国立大学）；Monash University（墨尔本大学）；Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology（香港科学大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
Multimodal Question Answering for Unified Information Extraction
The Ohio State University（俄亥俄州立大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Engineering, Westlake University（西湖大学工学院）
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis
Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；University of Cambridge（剑桥大学）
Shapley Value-based Contrastive Alignment for Multimodal Information Extraction
State Key Laboratory of Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science（计算机学院）；Peking University（北京大学）；Wangxuan Institute of Computer Technology（王选计算机技术研究所）
Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Code-MIE: A Code-style Model for Multimodal Information Extraction with Scene Graph and Entity Attribute Knowledge Enhancement
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University（航天信息安全部与可信计算重点实验室，教育部，武汉大学计算机科学与工程学院）；University of Oxford（牛津大学）；Wuhan Second Ship Design and Research Institute（武汉第二船舶设计研究所）；China United Network Communications Co., Ltd. Research Institute（中国联合网络通信有限公司研究院）
Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting
Emory University（埃默里大学）；Tongji University（同济大学）；The Hong Kong University of Science and Technology（香港科技大学）；Georgia Institute of Technology（佐治亚理工学院）；Northwestern University（西北大学）
SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis
National University of Singapore（新加坡国立大学）；Wuhan University（武汉大学）；The University of Auckland（奥克兰大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Nanyang Technological University（南洋理工大学）
Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment
School of Intelligent Systems Engineering, Sun Yat-sen University（1 智能系统工程学院，中山大学）；Alibaba Group（2 阿里巴巴集团）；Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology（3 广东省火灾科学与智能应急技术重点实验室）
InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictions
The Chinese University of Hong Kong（香港中文大学）；Hangzhou Institute of Medicine, CAS（杭州医学研究所，中国科学院）；Hangzhou Institute for Advanced Study, UCAS（杭州先进研究 institute，中国科学院大学）；University of Chinese Academy of Sciences（中国科学院大学）；Tianjin University（天津大学）
A Deep Learning Approach for Multimodal Deception Detection
A*STAR Artificial Intelligence Initiative (A*AI), Institute of High Performance Computing（新加坡科技研究局人工智能计划（A*AI），高性能计算研究所）；Centro de Investigacin en Computacin, IPN（墨西哥国立理工学院计算研究中心）；Temasek Laboratories, Nanyang Technological University（南洋理工大学淡马锡实验室）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
REMOTE: A Unified Multimodal Relation Extraction Framework with Multilevel Optimal Transport and Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
