Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
University of Science and Technology Beijing(北京科技大学);Beijing Engineering Research Center of Industrial Spectrum Imaging(北京工业光谱成像工程技术研究中心);Sun Yat-sen University(中山大学);Microsoft Research Asia(微软亚洲研究院)
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu
We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR$^2$ test-P split, 6.7% accuracy on SNLI-VE test split, respectively.
Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaning
Duncan of Jordanstone College of Art & Design (DJCAD), University of Dundee(邓迪大学邓迪艺术与设计学院(DJCAD));Guangzhou Institute of Science and Technology (GZIST)(广州科学技术研究院);Faculty of Arts, Xiamen University(厦门大学艺术学院);University of Birmingham(伯明翰大学)
Seeing and Solving Are Not Enough for Vision-Language Models
Sun Yat-sen University(中山大学);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科技大学);Institute of Information Engineering(信息工程研究所);Hunan University(湖南大学)
Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
Tsinghua University(清华大学);Peking University(北京大学);Fudan University(复旦大学);Microsoft Research Asia(微软亚洲研究院);Hong Kong University of Science and Technology(香港科技大学);Zhejiang University(浙江大学)
Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computing Technologies, RMIT University(皇家墨尔本理工学院计算技术学院)
Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);PCALab, VCIP, College of Computer Science, Nankai University(南开大学计算机学院PCALab, VCIP)
Seeing Red, Thinking Bad: Color Bias in Vision Language Models
National Institute of Advanced Industrial Science and Technology (AIST)(日本产业技术综合研究所);University of Tsukuba(筑波大学);University of Technology Nuremberg(纽伦堡工业大学);University of Oxford(牛津大学)
Can We Talk Models Into Seeing the World Differently?
Offenburg University(奥芬堡大学);University of Mannheim(曼海姆大学);Tübingen AI Center(蒂宾根人工智能中心);Goethe University Frankfurt(法兰克福大学);University of Siegen(锡根大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Google DeepMind(谷歌DeepMind);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)
Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
The University of Manchester(曼彻斯特大学);The University of Melbourne(墨尔本大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Edinburgh(爱丁堡大学);University of Southern California(南加州大学);Fudan University(复旦大学);University of Newcastle(纽卡斯尔大学)