Vision-Language Pre-training (VLP) models have shown remarkable performance on various downstream tasks. Their success heavily relies on the scale of pre-trained cross-modal datasets. However, the lack of large-scale datasets and benchmarks in Chinese hinders the development of Chinese VLP models and broader multilingual applications. In this work, we release a large-scale Chinese cross-modal dataset named Wukong, which contains 100 million Chinese image-text pairs collected from the web. Wukong aims to benchmark different multi-modal pre-training methods to facilitate the VLP research and community development. Furthermore, we release a group of models pre-trained with various image encoders (ViT-B/ViT-L/SwinT) and also apply advanced pre-training techniques into VLP such as locked-image text tuning, token-wise similarity in contrastive learning, and reduced-token interaction. Extensive experiments and a benchmarking of different downstream tasks including a new largest human-verified image-text test dataset are also provided. Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods. For the zero-shot image classification task on 10 datasets, $Wukong_{ViT-L}$ achieves an average accuracy of 73.03%. For the image-text retrieval task, it achieves a mean recall of 71.6% on AIC-ICC which is 12.9% higher than WenLan 2.0. Also, our Wukong models are benchmarked on downstream tasks with other variants on multiple datasets, e.g., Flickr8K-CN, Flickr-30K-CN, COCO-CN, et al. More information can be referred to: https://wukong-dataset.github.io/wukong-dataset/.
Joint Architecture and Knowledge Distillation in CNN for Chinese Text Recognition
University of Science and Technology of China（中国科学技术大学）；National Engineering Laboratory for Speech and Language Information Processing（语音及语言信息处理国家工程实验室）
Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University（阿里巴巴-南洋理工大学新加坡联合研究院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Zhejiang University（浙江大学）
