arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00300cs.CVcs.MM
收藏

迈向生成式视觉问答的灵活评估

Towards Flexible Evaluation for Generative Visual Question Answering

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学)

Huishan Ji, Qingyi Si, Zheng Lin, Weiping Wang

AI总结:

针对生成式 VQA 中精确匹配评估僵化的问题,本文构建 AVE 数据集与三项评估属性,并提出 SFVE 语义评估器,实验证明其显著优于现有语义评估方法且可跨架构泛化。

摘要:

在多模态大语言模型快速发展的过程中,一个关键要素是公平且准确地评估其多模态理解能力。尽管视觉问答(VQA)可以作为一个成熟的试验场,但 VQA 评估的局限,例如 Exact Match(精确匹配)这种不灵活的模式,已经阻碍 MLLM 展示其真实能力,并抑制内容丰富的回答。因此,本文提出使用基于语义的评估器,在 VQA 数据集上评估不受约束的开放式回答。由于 VQA 的特性使此类评估与传统的 Semantic Textual Similarity(STS,语义文本相似度)任务显著不同,为了系统分析各类评估器(包括基于 LLM 的评估器)的行为并比较其性能,我们提出三项关键属性,即 Alignment(对齐性)、Consistency(一致性)和 Generalization(泛化性),以及相应的数据集 Assessing VQA Evaluators(AVE)以促进分析。此外,本文基于 VQA 评估的独特特征,通过精细设计提出 Semantically Flexible VQA Evaluator(SFVE,语义灵活的 VQA 评估器)。实验结果验证了基于模型的 VQA 评估的可行性,以及所提评估器的有效性;该评估器大幅超越现有语义评估器。所提出的训练方案可泛化到类 BERT 编码器和 decoder-only LLM。

英文摘要:

Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different than the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we proposes three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation. Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑