LOVA3：学习视觉问答、提问与评估
LOVA3: Learning to Visual Question Answering, Asking and Assessment
AI总结：
提出 LOVA3 框架，通过 GenQA 与 EvalQA 任务及 EvalQABench 基准，让多模态大语言模型同时学习视觉问答、提问与评估，从而提升多模态理解和整体性能。
摘要：
回答问题、提出问题和评估问题是人类与生俱来的三项特质，对于理解世界和获取知识至关重要。通过增强这些能力，人类能够更有效地利用数据，从而获得更好的理解与学习效果。当前的多模态大语言模型（MLLMs）主要关注问答，往往忽视了提问和评估技能的全部潜力。受人类学习机制启发，我们提出了 LOVA3，这一创新框架名为“Learning tO Visual question Answering, Asking and Assessment”，旨在赋予 MLLMs 这些额外能力。我们的方法包括创建两个补充训练任务 GenQA 和 EvalQA，目标是在图像语境中培养提问和评估问题的技能。为发展提问能力，我们汇编了一套全面的多模态基础任务。在评估方面，我们引入了一个名为 EvalQABench 的新基准，包含 64,000 个训练样本（正样本和负样本各占一半）以及 5,000 个验证和测试样本。我们认为，赋予 MLLMs 回答、提出和评估问题的能力，将增强其多模态理解能力，并最终提升整体性能。为验证这一假设，我们使用 LOVA3 框架训练 MLLMs，并在一系列多模态数据集和基准上进行评估。结果显示性能持续提升，凸显了这些附加任务在培养 MLLMs 综合智能方面的关键作用。代码可在 https://github.com/showlab/LOVA3 获取。
英文摘要：
Question answering, asking, and assessment are three innate human traits crucial for understanding the world and acquiring knowledge. By enhancing these capabilities, humans can more effectively utilize data, leading to better comprehension and learning outcomes. Current Multimodal Large Language Models (MLLMs) primarily focus on question answering, often neglecting the full potential of questioning and assessment skills. Inspired by the human learning mechanism, we introduce LOVA3, an innovative framework named "Learning tO Visual question Answering, Asking and Assessment," designed to equip MLLMs with these additional capabilities. Our approach involves the creation of two supplementary training tasks GenQA and EvalQA, aiming at fostering the skills of asking and assessing questions in the context of images. To develop the questioning ability, we compile a comprehensive set of multimodal foundational tasks. For assessment, we introduce a new benchmark called EvalQABench, comprising 64,000 training samples (split evenly between positive and negative samples) and 5,000 validation and testing samples. We posit that enhancing MLLMs with the capabilities to answer, ask, and assess questions will enhance their multimodal comprehension, ultimately improving overall performance. To validate this hypothesis, we train MLLMs using the LOVA3 framework and evaluate them on a range of multimodal datasets and benchmarks. Our results demonstrate consistent performance gains, underscoring the critical role of these additional tasks in fostering comprehensive intelligence in MLLMs. The code is available at https://github.com/showlab/LOVA3.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
