This paper revisits visual representation in knowledge-based visual question answering (VQA) and demonstrates that using regional information in a better way can significantly improve the performance. While visual representation is extensively studied in traditional VQA, it is under-explored in knowledge-based VQA even though these two tasks share the common spirit, i.e., rely on visual input to answer the question. Specifically, we observe that in most state-of-the-art knowledge-based VQA methods: 1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent. Based on these observations, we propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model. The key motivation is that object regions and inherent relationship are important for knowledge-based VQA. We perform extensive experiments on the standard OK-VQA dataset and achieve new state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous state-of-the-art method by a large margin (+3.6%). We also conduct detailed analysis and show the necessity of regional information in different framework components for knowledge-based VQA. Code is publicly available at https://github.com/yzleroy/REVIVE.
In this paper, we propose a novel deep multi-level attention model to address inverse visual question answering. The proposed model generates regional visual and semantic features at the object level and then enhances them with the answer cue by using attention mechanisms. Two levels of multiple attentions are employed in the model, including the dual attention at the partial question encoding step and the dynamic attention at the next question word generation step. We evaluate the proposed model on the VQA V1 dataset. It demonstrates state-of-the-art performance in terms of multiple commonly …
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
United International University（国际联合大学）；BRAC University（布拉塔克大学）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
National and Local Joint Engineering Laboratory of Computer Aided Design, School of Software Engineering, Dalian University（大连大学软件工程学院计算机辅助设计国家地方联合工程实验室）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）
Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）；Remote Sensing Technology Institute（遥感技术研究所）
Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model w…
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
Capital One；SEACrowd；NAIST；ITB；UofT；KAIST；JAIST；Cardiff University；Independent；HKUST；SMU；Masakhane；University of Lagos；HK PolyU；Boston University；Columbia University；MBZUAI；Monash University；UCSB；Tokyo Tech；AI Singapore；NICT；Cohere；McGill；MILA；Canada CIFAR AI Chair；Ontario Tech
Describe Anything Model for Visual Question Answering on Text-rich Images
AI VIETNAM Lab（AI越南实验室）；Carnegie Mellon University（卡内基梅隆大学）；University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；University of Pittsburgh（匹兹堡大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Northwestern University（西北大学）
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture
University of Copenhagen（哥本哈根大学）；University of Waterloo（滑铁卢大学）；Comcast AI Technologies（康卡斯特人工智能技术公司）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Amsterdam（阿姆斯特丹大学）
Carnegie Mellon University（卡内基梅隆大学）；Google Research（谷歌研究院）；University of Washington（华盛顿大学）
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM r…
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University（京畿大学计算机学院）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；College of Computer Science and Information Engineering, Harbin Normal University（哈尔滨师范大学计算机科学与信息工程学院）；College of Professional and Continuing Education, The Hong Kong Polytechnic University（香港理工大学专业及继续教育学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
Baidu Research（百度研究院）；University of Technology Sydney（悉尼科技大学）
In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences ar…
Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
University of Central Florida（中央佛罗里达大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Columbia University（哥伦比亚大学）；Amazon Research（亚马逊研究）；University of California, Berkeley（加州大学伯克利分校）
Commonsense is defined as the knowledge that is shared by everyone. However, certain types of commonsense knowledge are correlated with culture and geographic locations and they are only shared locally. For example, the scenarios of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models' ability to understand cultural and geo-location-specif…
Arizona State University（亚利桑那州立大学）；Microsoft Corporation（微软公司）
Despite exciting progress in pre-training for visual-linguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student's detector while the f…
