Currently, image-text-driven multi-modal deep learning models have demonstrated their outstanding potential in many fields. In practice, tasks centered around facial images have broad application prospects. This paper presents \textbf{FaceCaption-15M}, a large-scale, diverse, and high-quality dataset of facial images accompanied by their natural language descriptions (facial image-to-text). This dataset aims to facilitate a study on face-centered tasks. FaceCaption-15M comprises over 15 million pairs of facial images and their corresponding natural language descriptions of facial features, making it the largest facial image-caption dataset to date. We conducted a comprehensive analysis of image quality, text naturalness, text complexity, and text-image relevance to demonstrate the superiority of FaceCaption-15M. To validate the effectiveness of FaceCaption-15M, we first trained a facial language-image pre-training model (FLIP, similar to CLIP) to align facial image with its corresponding captions in feature space. Subsequently, using both image and text encoders and fine-tuning only the linear layer, our FLIP-based models achieved state-of-the-art results on two challenging face-centered tasks. The purpose is to promote research in the field of face-related tasks through the availability of the proposed FaceCaption-15M dataset. All data, codes, and models are publicly available. https://huggingface.co/datasets/OpenFace-CQUPT/FaceCaption-15M
Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors
School of Computer Science, China University of Geosciences (Wuhan)(中国地质大学(武汉)计算机科学学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);School of Computer Science, Wuhan University(武汉大学计算机科学学院);School of Computing, Engineering and Mathematical Sciences, La Trobe University(拉筹伯大学计算科学、工程与数学科学学院)
Cross-Enhanced Multimodal Fusion of Eye-Tracking and Facial Features for Alzheimer's Disease Diagnosis
School of Control Science and Engineering, Shandong University(控制科学与工程学院,山东大学);Engineering Research Center of Intelligent Unmanned System, Ministry of Education(智能无人机系统工程研究中心,教育部);Department of Psychiatry, The Chinese University of Hong Kong(心理学系,香港中文大学);Department of Electronic Engineering, The Chinese University of Hong Kong(电子工程系,香港中文大学);AICARE Lab, Guangdong Medical University(AICARE实验室,广东医科大学);Department of Neurology, Shandong University of Traditional Chinese Medicine Affiliated Hospital(神经内科,山东中医药大学附属医院)
A Multimodal In Vitro Diagnostic Method for Parkinson's Disease Combining Facial Expressions and Behavioral Gait Data
School of Mathematics and Computer Sciences, Nanchang University, Nanchang, China(南昌大学数学与计算机科学学院);Yichun University, Yichun, China(宜春大学);Nanchang University Second Affiliated Hospital, Nanchang, China(南昌大学第二附属医院)
FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning
Tianjin University(天津大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Lappeenranta-Lahti University of Technology LUT(拉佩兰塔-拉赫蒂技术大学);Great Bay University(大湾大学);Southeast University(东南大学);Xi’an Jiaotong University(西安交通大学)
EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions
School of Computer Science, The University of Nottingham(诺丁汉大学计算机学院);University College London(伦敦大学学院);Department of Computer Science, University of East Anglia(东英吉利大学计算机科学系);Department of Electronic and Computer Engineering, Brunel University London(布鲁内尔大学伦敦电子与计算机工程系);Department of Computer Science, University of Oxford(牛津大学计算机科学系)
When Words Smile: Generating Diverse Emotional Facial Expressions from Text
Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));University of Macau(澳门大学);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学)
High-Fidelity Facial Albedo Estimation via Texture Quantization
University of Technology Sydney(悉尼科技大学);Shanghai Jiao Tong University(上海交通大学);DeepGlint(旷视科技(北京));Insightface;Zhejiang University(浙江大学);Imperial College London(伦敦帝国学院)
RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing
Beihang University(北京航空航天大学);Pengcheng Laboratory(鹏城实验室);Shanxi University of Finance and Economics(山西财经大学);North China University of Technology(北方工业大学);Cardiff University(卡迪夫大学);Beijing Institute of Technology(北京理工大学)
FreqFLD: Towards All-in-One Facial Landmark Detection via Frequency Modulation
China Three Gorges University(三峡大学);Zhongnan University of Economics and Law(中南财经政法大学);National Institute of Natural Hazards, Ministry of Emergency Management of China(中国应急管理部国家自然灾害防治研究院);Yunnan University(云南大学);Wuhan University(武汉大学);Nanyang Technological University(南洋理工大学)
Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis
State University of New York at Binghamton(纽约州立大学布法罗分校);Choate Rosemary Hall(乔斯-罗丝玛丽高中);Rensselaer Polytechnic Institute(拉特格斯理工学院);Ward Melville High School(沃德梅尔维尔高中);University of Pittsburgh(匹兹堡大学)
Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
Computational Biomedicine Laboratory of the Foundation for Research and Technology (FORTH)(希腊国家研究与技术基金会计算生物医学实验室);Department of Electrical & Computer Engineering Hellenic Mediterranean University(希腊地中海大学电气与计算机工程系)
INRIA Sophia Antipolis - Méditerranée(法国国家信息与自动化研究所索菲亚·安提波利斯-地中海研究中心);Birla Institute of Technology and Science, Pilani(皮拉尼·博拉理工学院);Université Côte d’Azur(蔚蓝海岸大学)
AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
Westlake University(西湖大学);Zhejiang University(浙江大学);Tiangong University(天工大学);Hunan University(湖南大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology(国防科技大学系统工程学院大数据与决策实验室)
FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing
College of Computer Science and Artificial Intelligence, Wenzhou University(温州大学计算机与人工智能学院);Memorial University of Newfoundland(纽芬兰纪念大学);State Key Laboratory of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室)