Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different than the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we proposes three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation. Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM.
WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering
Zhejiang University(浙江大学);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);School of Engineering, Westlake University(西湖大学工程学院)
VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction
1 Academy for Engineering;Technology, Fudan University, Shanghai, China;2 Institute of Metaverse \& Intelligent Medicine, Fudan University, Shanghai, China;3 Engineering Research Center of AI;4 Jilin Provincial Key Laboratory of Intelligence Science
Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification
School of Computing and Information Technology, Great Bay University(大湾大学计算机与信息学院);Dongguan Key Laboratory for Intelligence and Information Technology(东莞智能与信息科技重点实验室);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Computer Science, Nankai University(南开大学计算机学院);School of mathematics, Sun Yat-sen University(中山大学数学学院)
Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models
University of Hong Kong(香港大学);University College London(伦敦大学学院);University of Alberta(阿尔伯塔大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Zhejiang University(浙江大学)