arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02233cs.CLcs.AIcs.LG
收藏

合成多模态问题生成

Synthetic Multimodal Question Generation

C3 AI(C3 AI公司);Connectly AI(Connectly AI公司);Carnegie Mellon University(卡内基梅隆大学)

Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Pakazad, Tongshuang Wu, Graham Neubig

AI总结:

针对多模态检索增强生成(MMRAG)评估缺乏匹配风格与模态的高质量数据集的问题,提出SMMQG合成数据生成框架,生成的数据集可有效评估模型,数据质量与众包基准相当。

摘要:

多模态检索增强生成(MMRAG)是针对多模态文档进行问答的强大方法。评估MMRAG的一个关键挑战是,缺乏与目标问题风格和模态相匹配的高质量数据集。鉴于此,我们提出了SMMQG,一种合成数据生成框架。SMMQG利用检索器、大语言模型(LLM)和大多模态模型(LMM)之间的交互,直接从多模态文档生成问答对,且生成的问题符合指定的风格和模态。我们使用SMMQG基于Wikipedia文档生成了包含1024个问题的MMRAG数据集,并用该数据集评估了前沿模型,揭示了只有通过风格和模态特定的评估数据才能获得的模型性能洞见。随后,我们通过人类研究衡量了SMMQG生成数据的质量。我们发现,SMMQG生成的合成数据质量与众包基准MMQA的质量相当,且使用这两个数据集得到的下游评估结果高度一致。

英文摘要:

Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and modalities of interest. In light of this, we propose SMMQG, a synthetic data generation framework. SMMQG leverages interplay between a retriever, large language model (LLM) and large multimodal model (LMM) to generate question and answer pairs directly from multimodal documents, with the questions conforming to specified styles and modalities. We use SMMQG to generate an MMRAG dataset of 1024 questions over Wikipedia documents and evaluate state-of-the-art models using it, revealing insights into model performance that are attainable only through style- and modality-specific evaluation data. Next, we measure the quality of data produced by SMMQG via a human study. We find that the quality of SMMQG-generated synthetic data is on par with the quality of the crowdsourced benchmark MMQA and that downstream evaluation results using both datasets strongly concur.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑