arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.08515cs.CVcs.AI
收藏

15M多模态人脸图像-文本数据集

15M Multimodal Facial Image-Text Dataset

College of Computer Science and Technology, Chongqing University of Posts and Telecommunications(重庆邮电大学计算机科学与技术学院)

Dawei Dai, YuTang Li, YingGe Liu, Mingming Jia, Zhang YuanHui, Guoyin Wang

AI总结:

本研究发布了目前规模最大的人脸图像-文本数据集FaceCaption-15M(含1500万余对样本),通过多维度分析验证其质量,基于该数据集训练的FLIP模型在两项人脸任务上取得最优结果,将公开资源以推动相关领域研究。

摘要:

当前,由图像-文本驱动的多模态深度学习模型已在诸多领域展现出卓越潜力。在实际应用中,以人脸图像为核心的任务拥有广阔的应用前景。本文提出了**FaceCaption-15M**,这是一个大规模、多样化且高质量的人脸图像数据集,配套有对应的自然语言描述(人脸图像转文本),旨在推动以人脸为核心的各类任务的研究。FaceCaption-15M包含超过1500万组人脸图像及其对应的面部特征自然语言描述对,是目前规模最大的人脸图像-字幕数据集。我们从图像质量、文本自然度、文本复杂度以及图文相关性四个维度开展了全面分析,以证明FaceCaption-15M的优越性。为验证该数据集的有效性,我们首先训练了一个人脸语言-图像预训练模型(FLIP,类似CLIP),实现人脸图像与对应字幕在特征空间中的对齐。随后,借助图像编码器与文本编码器,仅微调线性层,我们基于FLIP的模型就在两项极具挑战性的人脸核心任务上取得了当前最优(state-of-the-art)结果。本研究的目标是通过公开FaceCaption-15M数据集,推动人脸相关任务领域的研究发展。所有数据、代码与模型均已公开,可访问https://huggingface.co/datasets/OpenFace-CQUPT/FaceCaption-15M获取。

英文摘要:

Currently, image-text-driven multi-modal deep learning models have demonstrated their outstanding potential in many fields. In practice, tasks centered around facial images have broad application prospects. This paper presents \textbf{FaceCaption-15M}, a large-scale, diverse, and high-quality dataset of facial images accompanied by their natural language descriptions (facial image-to-text). This dataset aims to facilitate a study on face-centered tasks. FaceCaption-15M comprises over 15 million pairs of facial images and their corresponding natural language descriptions of facial features, making it the largest facial image-caption dataset to date. We conducted a comprehensive analysis of image quality, text naturalness, text complexity, and text-image relevance to demonstrate the superiority of FaceCaption-15M. To validate the effectiveness of FaceCaption-15M, we first trained a facial language-image pre-training model (FLIP, similar to CLIP) to align facial image with its corresponding captions in feature space. Subsequently, using both image and text encoders and fine-tuning only the linear layer, our FLIP-based models achieved state-of-the-art results on two challenging face-centered tasks. The purpose is to promote research in the field of face-related tasks through the availability of the proposed FaceCaption-15M dataset. All data, codes, and models are publicly available. https://huggingface.co/datasets/OpenFace-CQUPT/FaceCaption-15M

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑