MMHQA-ICL:面向文本、表格和图像的混合问答的多模态上下文学习
MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images
AI总结:
针对包含文本、表格和图像的混合问答任务,提出MMHQA-ICL框架,通过异构数据检索器、图像描述模块及类型特定的上下文学习策略,首次采用端到端LLM提示方法,在MultimodalQA数据集少样本设置下达到最优性能。
摘要:
在现实世界中,知识通常以多模态和异构的形式存在。处理包含文本、表格和图像等混合数据类型的问答任务是一项具有挑战性的任务(MMHQA)。近期,随着大语言模型(LLM)的兴起,上下文学习(ICL)已成为解决QA问题的最流行方式。我们提出了MMHQA-ICL框架来解决这些问题,该框架包含更强的异构数据检索器和图像描述模块。最重要的是,我们为MMHQA提出了一种类型特定的上下文学习策略,使LLM能够在此任务中发挥其强大的性能。我们是首个在此任务中使用端到端LLM提示方法的研究。实验结果表明,我们的框架优于所有基线以及在完整数据集上训练的方法,在MultimodalQA数据集的少样本设置下取得了最先进的结果。
英文摘要:
In the real world, knowledge often exists in a multimodal and heterogeneous form. Addressing the task of question answering with hybrid data types, including text, tables, and images, is a challenging task (MMHQA). Recently, with the rise of large language models (LLM), in-context learning (ICL) has become the most popular way to solve QA problems. We propose MMHQA-ICL framework for addressing this problems, which includes stronger heterogeneous data retriever and an image caption module. Most importantly, we propose a Type-specific In-context Learning Strategy for MMHQA, enabling LLMs to leverage their powerful performance in this task. We are the first to use end-to-end LLM prompting method for this task. Experimental results demonstrate that our framework outperforms all baselines and methods trained on the full dataset, achieving state-of-the-art results under the few-shot setting on the MultimodalQA dataset.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。