In this paper, we make a simple observation that questions about images often contain premises - objects and relationships implied by the question - and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.
PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);ModelBest Inc.(ModelBest公司);State Key Lab. of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Institute for AI, Tsinghua University(清华大学人工智能研究院)
Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering
PAIQ: Patch-Aligned Semantic Injection via Residual Rotation
Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University(浙江大学ZJU-UIUC学院);University of Washington(华盛顿大学);Zhejiang University School of Medicine Sir Run Run Shaw Hospital(浙江大学医学院附属邵逸夫医院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
MOSMOS: Multi-organ segmentation facilitated by medical report supervision
Academy for Engineering and Technology, Fudan University(复旦大学工程技术研究院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Changhai Hospital, Second Military Medical University (Naval Medical University)(海军军医大学长海医院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程学系)
Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes
GADE Union (Global AI Data Experts Union)(GADE联盟(全球人工智能数据专家联盟));Shanghai Jiao Tong University(上海交通大学);Nanyang Technological University(南洋理工大学);New York University(纽约大学);Cambridge University(剑桥大学);The University of Hong Kong(香港大学)
EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis
The Hong Kong Polytechnic University(香港理工大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院);Clemson University(克莱姆森大学);Centre for Eye Research Australia, Royal Victorian Eye and Ear Hospital(澳大利亚眼科研究中心,皇家维多利亚眼耳医院);Zhejiang University(浙江大学);Bright Eye Hospital(亮睛眼科医院);Guangdong Academy of Medical Sciences, Guangdong Provincial People's Hospital(广东省医学科学院,广东省人民医院);Capital Medical University(首都医科大学);Beijing Tongren Hospital(北京同仁医院);Centre for Eye and Vision Research (CEVR)(眼科与视觉研究中心)