利用分类数据集及其语义层次对视觉-语言模型进行开放式VQA基准测试
Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
AI总结：
本文提出一种基于分类数据集及其语义层次的新型开放式VQA基准，通过自动生成后续问题实现细粒度评估，并比较NLP与LLM指标，为视觉-语言模型能力评估奠定更精确的基础。
摘要：
对文本生成式视觉-语言模型的评估是一项具有挑战性但至关重要的工作。通过解决现有视觉问答（VQA）基准的局限性并提出创新的评估方法，我们的研究旨在加深对这些模型能力的理解。我们提出了一种基于知名视觉分类数据集的新型VQA基准，该基准允许对文本生成式视觉-语言模型进行细粒度评估，并与判别式视觉-语言模型进行比较。为了改进对细粒度分类任务中粗略答案的评估，我们建议利用标签空间的语义层次来自动生成关于真实类别的后续问题。最后，我们比较了传统NLP指标和基于LLM的指标在给定真实答案情况下评估模型预测的问题。我们进行了一项人类评估研究，并以此为基础确定最终指标。我们将基准应用于一系列视觉-语言模型，并展示了它们在物体、动作和属性分类能力上的详细比较。我们的贡献旨在为更精确、更有意义的评估奠定基础，促进视觉-语言建模这一令人兴奋的领域取得有针对性的进展。
英文摘要：
The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
