arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09269cs.CVcs.AIcs.CLcs.LG
收藏

面向跨任务、跨领域、跨知识类型视觉问答的视觉语言模型选择引导

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

Georgia Institute of Technology(佐治亚理工学院);Meta AI;Amazon GenAI(亚马逊生成式人工智能部门)

Neelabh Sinha, Vinija Jain, Aman Chadha

AI总结:

针对实际场景下视觉语言模型评估选型难的问题,本文提出带多维度标注的VQA360数据集和基于GPT-4o的GoEval评估指标,验证了不同模型各有优劣,该框架可扩展至其他任务。

摘要:

视觉问答(VQA)已成为影响用户体验的关键因素,尤其是在视觉语言模型(VLM)的泛化能力得到提升之后。但在实际场景中,使用标准化框架针对应用需求评估VLM仍具挑战性。本文旨在通过一个端到端框架解决该问题。我们提出VQA360——一个源自现有VQA基准的新型数据集,标注有任务类型、应用领域和知识类型,可用于全面评估。我们还推出了GoEval,这是一种基于GPT-4o开发的多模态评估指标,与人类判断的相关系数达56.71%。我们对前沿VLM的实验表明,没有单一模型能在所有场景下都表现优异,因此做出正确选择是关键的设计决策。Gemini-1.5-Pro和GPT-4o-mini等专有模型通常优于其他模型,但InternVL-2-8B和CogVLM-2-Llama-3-19B等开源模型也展现出有竞争力的优势,同时还具备额外的长处。我们的框架还可扩展到其他任务。

英文摘要:

Visual Question-Answering (VQA) has become key to user experience, particularly after improved generalization capabilities of Vision-Language Models (VLMs). But evaluating VLMs for an application requirement using a standardized framework in practical settings is still challenging. This paper aims to solve that using an end-to-end framework. We present VQA360 - a novel dataset derived from established VQA benchmarks, annotated with task types, application domains, and knowledge types, for a comprehensive evaluation. We also introduce GoEval, a multimodal evaluation metric developed using GPT-4o, achieving a correlation factor of 56.71% with human judgments. Our experiments with state-of-the-art VLMs reveal that no single model excels universally, thus, making a right choice a key design decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally outperform others, but open-source models like InternVL-2-8B and CogVLM-2-Llama-3-19B also demonstrate competitive strengths, while providing additional advantages. Our framework can also be extended to other tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑