UKnow：一种面向推理与视觉-语言预训练的统一知识协议及多模态知识图谱数据集
UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training
AI总结：
本文提出统一知识协议 UKnow，将视觉与语言数据知识划分为五类单元并构建多模态知识图谱，基于国际新闻构建含 138 万余节点和 367 万余三元组的大规模数据集，在 4 个基准上验证了其对常识推理和视觉-语言预训练的提升作用。
摘要：
本文提出了一种名为 UKnow 的统一知识协议，旨在从数据视角促进基于知识的研究。我们特别关注视觉和语言两种模态，将数据知识划分为五种单元类型，即图像内、文本内、跨图像、跨文本和图像-文本，并建立了一套高效流程，用于从任意数据集合中构建多模态知识图谱。由于知识图谱天然蕴含逻辑信息，与常用的图像-文本对相比，以 UKnow 格式组织数据集为数据使用开辟了更多可能性。遵循 UKnow 协议，我们从国际公共新闻中收集了一个大规模多模态知识图谱数据集，该数据集包含 1,388,568 个节点（其中 571,791 个与视觉相关）和 3,673,817 个三元组。该数据集还标注了丰富的事件标签，包括 11 个粗粒度标签和 9,185 个细粒度标签。在 4 个基准上的实验表明，UKnow 凭借其统一的知识组织形式，能够以单一数据集支持常识推理并提升视觉-语言预训练。数据集下载方式见附录。
英文摘要：
This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. Particularly focusing on visual and linguistic modalities, we categorize data knowledge into five unit types, namely, in-image, in-text, cross-image, cross-text, and image-text, and set up an efficient pipeline to help construct the multimodal knowledge graph from any data collection. Thanks to the logical information naturally contained in knowledge graph, organizing datasets under UKnow format opens up more possibilities of data usage compared to the commonly used image-text pairs. Following UKnow protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset that consists of 1,388,568 nodes (with 571,791 vision-related ones) and 3,673,817 triplets. The dataset is also annotated with rich event tags, including 11 coarse labels and 9,185 fine labels. Experiments on 4 benchmarks demonstrate the potential of UKnow in supporting common-sense reasoning and boosting vision-language pre-training with a single dataset, benefiting from its unified form of knowledge organization. See Appendix to download the dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
