面向利用大视觉语言模型高效且鲁棒地生成VQA-NLE数据
Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models
AI总结：
针对现有VQA-NLE数据集构建依赖高成本人工标注的问题，本研究提出利用大视觉语言模型结合先进提示技术生成合成数据的方法，速度最高快20倍且质量接近人工标注，还验证了视觉提示的提升作用，为多模态NLE数据自动生成提供了高效鲁棒的方案。
摘要：
自然语言解释（Natural Language Explanation, NLE）旨在通过提供详细、人类友好的自然语言解释来阐明决策过程，它借助语言模型帮助揭开大视觉语言模型（Large Vision-Language Models, LVLMs）决策过程的神秘面纱。现有的视觉问答自然语言解释（Vision Question-Answering with Natural Language Explanation, VQA-NLE）数据集构建方法虽能提供解释，但严重依赖人工标注，耗时且成本高昂。本研究提出一种新方法，利用LVLMs高效生成高质量的合成VQA-NLE数据集。通过对合成数据的评估，我们展示了先进的提示技术如何助力生成高质量的VQA-NLE数据。研究结果表明，该方法的速度比人工标注最高快20倍，定性指标仅出现极小幅度下降，具备接近人工标注数据的鲁棒质量。此外，我们还发现加入视觉提示能显著提升文本生成的相关性。本研究为更高效、鲁棒的多模态NLE数据自动生成铺平了道路，为该问题提供了一种有前景的解决方案。
英文摘要：
Natural Language Explanation (NLE) aims to elucidate the decision-making process by providing detailed, human-friendly explanations in natural language. It helps demystify the decision-making processes of large vision-language models (LVLMs) through the use of language models. While existing methods for creating a Vision Question-Answering with Natural Language Explanation (VQA-NLE) datasets can provide explanations, they heavily rely on human annotations that are time-consuming and costly. In this study, we propose a novel approach that leverages LVLMs to efficiently generate high-quality synthetic VQA-NLE datasets. By evaluating our synthetic data, we showcase how advanced prompting techniques can lead to the production of high-quality VQA-NLE data. Our findings indicate that this proposed method achieves up to 20x faster than human annotation, with only a minimal decrease in qualitative metrics, achieving robust quality that is nearly equivalent to human-annotated data. Furthermore, we show that incorporating visual prompts significantly enhances the relevance of text generation. Our study paves the way for a more efficient and robust automated generation of multi-modal NLE data, offering a promising solution to the problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
