Fact-based Visual Question Answering (FVQA), a challenging variant of VQA, requires a QA-system to include facts from a diverse knowledge graph (KG) in its reasoning process to produce an answer. Large KGs, especially common-sense KGs, are known to be incomplete, i.e., not all non-existent facts are always incorrect. Therefore, being able to reason over incomplete KGs for QA is a critical requirement in real-world applications that has not been addressed extensively in the literature. We develop a novel QA architecture that allows us to reason over incomplete KGs, something current FVQA state-of-the-art (SOTA) approaches lack due to their critical reliance on fact retrieval. We use KG Embeddings, a technique widely used for KG completion, for the downstream task of FVQA. We also employ a new image representation technique we call 'Image-as-Knowledge' to enable this capability, alongside a simple one-step CoAttention mechanism to attend to text and image during QA. Our FVQA architecture is faster during inference time, being O(m), as opposed to existing FVQA SOTA methods which are O(N log N), where m = number of vertices, N = number of edges = O(m^2). KG embeddings are shown to hold complementary information to word embeddings: a combination of both metrics permits performance comparable to SOTA methods in the standard answer retrieval task, and significantly better (26% absolute) in the proposed missing-edge reasoning task.
Seeing Red, Thinking Bad: Color Bias in Vision Language Models
National Institute of Advanced Industrial Science and Technology (AIST)(日本产业技术综合研究所);University of Tsukuba(筑波大学);University of Technology Nuremberg(纽伦堡工业大学);University of Oxford(牛津大学)
AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(台湾国立台湾大学通信工程研究所);Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan(台湾国立台湾大学人工智能研究中心)
Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
University of Oregon Eugene OR USA;Pacific Northwest National Laboratory Richland WA USA;Adobe Research San Jose CA USA;Texas A\&M University College Station TX USA;Rensselaer Polytechnic Institute Troy NY USA;University of Oregon;Pacific Northwest National Laboratory;Adobe Research;Texas A\&M University;Rensselaer Polytechnic Institute