An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge repositories. However, many existing knowledge bases only cover limited domains, or suffer from noisy data, and most of all are typically hard to integrate into neural language pipelines. To fill this gap, we release VisualSem: a high-quality knowledge graph (KG) which includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations. We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG. This multi-modal retrieval model can be integrated into any (neural network) model pipeline. We encourage the research community to use VisualSem for data augmentation and/or as a source of grounding, among other possible uses. VisualSem as well as the multi-modal retrieval models are publicly available and can be downloaded in this URL: https://github.com/iacercalixto/visualsem
Johns Hopkins University（约翰斯·霍普金斯大学）；New York University（纽约大学）；University of Amsterdam（阿姆斯特丹大学）；Rond Consulting（朗德咨询公司）；Amazon Alexa AI（亚马逊Alexa人工智能部门）；Amsterdam UMC（阿姆斯特丹大学医学中心）
We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG using a dense index. Given (possibly multilingual) downstream task data, e.g., sentences in German, we retrieve entities from the KG and use their multimodal representations to improve downstream task performance. We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn enti…
Deep learning models suffer from opaqueness. For Convolutional Neural Networks (CNNs), current research strategies for explaining models focus on the target classes within the associated training dataset. As a result, the understanding of hidden feature map activations is limited by the discriminative knowledge gleaned during training. The aim of our work is to explain and expand CNNs models via the mirroring or alignment of CNN to an external knowledge base. This will allow us to give a semantic context or label for each visual feature. We can match CNN feature activations to nodes in our ext…
If a robot is supposed to roam an environment and interact with objects, it is often necessary to know all possible objects in advance, so that a database with models of all objects can be generated for visual identification. However, this constraint cannot always be fulfilled. Due to that reason, a model based object recognition cannot be used to guide the robot's interactions. Therefore, this paper proposes a system that analyzes features of encountered objects and then uses these features to compare unknown objects to already known ones. From the resulting similarity appropriate actions can…
Tsinghua University（清华大学）；University of Oxford（牛津大学）；Tencent（腾讯）；Peng Cheng Laboratory（鹏城实验室）；The University of Manchester（曼彻斯特大学）
Entity alignment (EA) for knowledge graphs (KGs) plays a critical role in knowledge engineering. Existing EA methods mostly focus on utilizing the graph structures and entity attributes (including literals), but ignore images that are common in modern multi-modal KGs. In this study we first constructed Multi-OpenEA -- eight large-scale, image-equipped EA benchmarks, and then evaluated some existing embedding-based methods for utilizing images. In view of the complementary nature of visual modal information and logical deduction, we further developed a new multi-modal EA method named LODEME usi…
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficie…
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）
Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong（香港中文大学深圳大数据研究院）；Sun Yat-sen University（中山大学）；Pazhou Lab（琶洲实验室）
Medical vision-and-language pre-training (Med-VLP) has received considerable attention owing to its applicability to extracting generic vision-and-language representations from medical images and texts. Most existing methods mainly contain three elements: uni-modal encoders (i.e., a vision encoder and a language encoder), a multi-modal fusion module, and pretext tasks, with few studies considering the importance of medical domain expert knowledge and explicitly exploiting such knowledge to facilitate Med-VLP. Although there exist knowledge-enhanced vision-and-language pre-training (VLP) method…
VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark
University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；Nanjing University（南京大学）
VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Washington（华盛顿大学）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究所）；Chinese Academy of Sciences（中国科学院）；State Key Laboratory of Ophthalmology（眼科学国家重点实验室）；Zhongshan Ophthalmic Center（中山眼科中心）；Sun Yat-sen University（中山大学）；Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science（广东省眼科学与视觉科学重点实验室）；Guangdong Provincial Clinical Research Center for Ocular Diseases（广东省眼科临床研究中心）；Department of Bioengineering（生物工程系）；Department of Radiology（放射科）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Amazon Alexa AI（亚马逊Alexa人工智能部门）
A key goal for the advancement of AI is to develop technologies that serve the needs not just of one group but of all communities regardless of their geographical region. In fact, a significant proportion of knowledge is locally shared by people from certain regions but may not apply equally in other regions because of cultural differences. If a model is unaware of regional characteristics, it may lead to performance disparity across regions and result in bias against underrepresented groups. We propose GIVL, a Geographically Inclusive Vision-and-Language Pre-trained model. There are two attri…
We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, thes…
Anticipating future actions in a video is useful for many autonomous and assistive technologies. Most prior action anticipation work treat this as a vision modality problem, where the models learn the task information primarily from the video features in the action anticipation datasets. However, knowledge about action sequences can also be obtained from external textual data. In this work, we show how knowledge in pretrained language models can be adapted and distilled into vision-based action anticipation models. We show that a simple distillation technique can achieve effective knowledge tr…
Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Tencent（腾讯）；Renmin University of China（中国人民大学）；Johns Hopkins University（约翰霍普金斯大学）
Beihang University（北京航空航天大学）；ETH Zurich（苏黎世联邦理工学院）；Bytedance AI Lab（字节跳动人工智能实验室）
Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and latency constraints. In this work, we introduce a distilling then pruning framework to compress large vision-language models into smaller, faster, and more accurate ones. We first shrink the size of a pre-trained large VLM and apply knowledge distillation in the vision-language pre-training stage to obt…
Hangzhou Dianzi University（杭州电子科技大学）；Alibaba Group（阿里巴巴集团）；Hefei University of Technology（合肥工业大学）
Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cross- and i…
Vision-and-language tasks are gaining popularity in the research community, but the focus is still mainly on English. We propose a pipeline that utilizes English-only vision-language models to train a monolingual model for a target language. We propose to extend OSCAR+, a model which leverages object tags as anchor points for learning image-text alignments, to train on visual question answering datasets in different languages. We propose a novel approach to knowledge distillation to train the model in other languages using parallel sentences. Compared to other models that use the target langua…
MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；UMass Amherst（马萨诸塞大学阿默斯特分校）；Tsinghua University（清华大学）
Large pre-trained vision and language models have demonstrated remarkable capacities for various tasks. However, solving the knowledge-based visual reasoning tasks remains challenging, which requires a model to comprehensively understand image content, connect the external world knowledge, and perform step-by-step reasoning to answer the questions correctly. To this end, we propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. The see stage scans the image and grounds the v…
Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
Morgan Stanley（摩根士丹利）；Clemson University（克莱姆森大学）；Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Arizona（亚利桑那大学）；University of Notre Dame（圣母大学）
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
Boston University（波士顿大学）；University of Amsterdam（阿姆斯特丹大学）；Umeå University（乌梅大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；TTIC（TTIC研究中心）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models
Ben Gurion University（本古里安大学）；Bar-Ilan University（巴伊兰大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Google Research（谷歌研究院）；Tel Aviv University（特拉维夫大学）
The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a different question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these …
Chalmers University of Technology（查尔姆斯理工大学）；Recorded Future
Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of mem…
Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed m…
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
Georgetown University（乔治城大学）；University of Southern California（南加州大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Arizona State University（亚利桑那州立大学）
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（智能视觉与成像上海工程研究中心）；Institute for Intelligent Computing, Alibaba Group（智能计算研究院，阿里巴巴集团）
