While there is much excitement about the potential of large multimodal models (LMM), a comprehensive evaluation is critical to establish their true capabilities and limitations. In support of this aim, we evaluate two state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering dataset sourced from an authentic online question answering community. We conduct fine-grained analysis by generating seven types of metadata for nearly 2,000 visual questions, such as image type and the required image processing capabilities. Our zero-shot performance analysis highlights the types of questions that are most challenging for both models, including questions related to "puzzling" topic, with "Identification" user intention, with "Sheet Music" image type, or labeled as "hard" by GPT-4.
SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery
National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；National Institute of Technology, Tiruchirappalli（蒂鲁吉拉伯利国立理工学院）；Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering, Chinese University of Hong Kong（香港中文大学信兴高等工程研究所）
Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery（高端工程机械智能制造国家重点实验室）；Department of Electrical and Computer Engineering, Sungkyunkwan University（成均馆大学电气与计算机工程系）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
New York University, New York, USA（纽约大学）；Tsinghua University, Beijing, China（清华大学）；Columbia University, New York, USA（哥伦比亚大学）；University of Michigan, Ann Arbor, USA（密歇根大学）
South China University of Technology（华南理工大学）；Johns Hopkins University（约翰霍普金斯大学）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院（教育部人工智能重点实验室））；Huawei Technologies（华为技术有限公司）
An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
Faculty of Computing and Information Technology, Sohar University（苏哈尔大学计算与信息技术学院）；Foundation University School of Sciences and Technology（基础大学科学与技术学院）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）
Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
Institute of Medical Research, Northwestern Polytechnical University（西北工业大学医学研究院）；School of Computing, University of Georgia（佐治亚大学计算学院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Lingang Laboratory（临港实验室）
