arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.10086cs.AI
收藏

ARMADA:基于属性的多模态数据增强

ARMADA: Attribute-Based Multimodal Data Augmentation

Texas A&M University(德克萨斯农工大学);University of Illinois(伊利诺伊大学);University of California(加利福尼亚大学)

Xiaomeng Jin, Jeonghwan Kim, Yu Zhou, Kuan-Hao Huang, Te-Lin Wu, Nanyun Peng, Heng Ji

AI总结:

针对现有多模态数据增强框架存在图文语义不一致、生成图像不真实的问题,提出基于知识引导调整实体视觉属性的ARMADA框架,可生成高质量增强数据,有效提升多模态语言模型下游任务性能。

摘要:

在多模态语言模型(MLMs)中,为微调与对齐手动标注高质量图文对数据的成本极高。现有多模态数据增强框架虽提出了多种图文对增强方式,但要么存在图文语义不一致的问题,要么生成的图像不真实,导致与真实样本间存在知识鸿沟。为解决这些问题,我们提出基于属性的多模态数据增强方法ARMADA,这是一种通过知识引导调整提及实体视觉属性的新型多模态数据增强方法。具体而言,我们从原始文本数据中提取实体及其视觉属性,随后在知识库(KBs)与大语言模型(LLMs)的引导下,搜索这些视觉属性的替代取值。接着我们利用图像编辑模型,根据提取的属性对图像进行编辑。ARMADA是一种新型多模态数据生成框架,具备三大特点:(i)从符号化知识库中提取基于知识的属性,以生成语义一致且各具差异的图文对;(ii)利用知识库层级结构中的邻近实体,生成不同类别但视觉相似的图像;(iii)借助大语言模型的常识知识调整背景等辅助视觉属性,使原始实体的表征更具鲁棒性。我们在四项下游任务上的实验结果表明,该框架能够生成高质量数据并提升模型性能。这也凸显了利用外部知识代理来增强可解释性与现实世界 grounding 能力的必要性。

英文摘要:

In Multimodal Language Models (MLMs), the cost of manually annotating high-quality image-text pair data for fine-tuning and alignment is extremely high. While existing multimodal data augmentation frameworks propose ways to augment image-text pairs, they either suffer from semantic inconsistency between texts and images, or generate unrealistic images, causing knowledge gap with real world examples. To address these issues, we propose Attribute-based Multimodal Data Augmentation (ARMADA), a novel multimodal data augmentation method via knowledge-guided manipulation of visual attributes of the mentioned entities. Specifically, we extract entities and their visual attributes from the original text data, then search for alternative values for the visual attributes under the guidance of knowledge bases (KBs) and large language models (LLMs). We then utilize an image-editing model to edit the images with the extracted attributes. ARMADA is a novel multimodal data generation framework that: (i) extracts knowledge-grounded attributes from symbolic KBs for semantically consistent yet distinctive image-text pair generation, (ii) generates visually similar images of disparate categories using neighboring entities in the KB hierarchy, and (iii) uses the commonsense knowledge of LLMs to modulate auxiliary visual attributes such as backgrounds for more robust representation of original entities. Our empirical results over four downstream tasks demonstrate the efficacy of our framework to produce high-quality data and enhance the model performance. This also highlights the need to leverage external knowledge proxies for enhanced interpretability and real-world grounding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑