探索用于面部属性识别的视觉语言模型：情绪、种族、性别与年龄
Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age
AI总结：
该研究探索视觉语言模型在面部属性识别中的应用，提出微调模型FaceScanPaliGemma与多人物识别模型FaceScanGPT，在多数据集上取得优于传统方法与现有最优方案的性能。
摘要：
种族、性别、年龄、情绪等面部属性识别技术具备多类应用场景，例如安防监控、广告内容制作、情感分析，以及人口趋势与社会行为研究。由于人类面部属性的复杂性，基于图像分析人口统计学特征与分析面部表情均存在诸多挑战。传统方法采用卷积神经网络（CNNs）及各类其他深度学习技术，在大规模标注图像数据集上训练。尽管这类方法已展现出有效性能，但仍存在进一步提升的空间。本文提出利用视觉语言模型（VLMs），包括生成式预训练Transformer（GPT）、GEMINI、大语言视觉助手（LLAVA）、PaliGemma以及Microsoft Florence2，从包含人脸的图像中识别种族、性别、年龄、情绪等面部属性。研究使用FairFace、AffectNet、UTKFace等多个数据集对方案进行评估。结果表明，VLMs即便未超越传统技术，也具备相当的竞争力。此外，本文提出“FaceScanPaliGemma”——一种经过微调的PaliGemma模型，用于种族、性别、年龄与情绪识别。结果显示，其在种族、性别、年龄组、情绪分类任务上的准确率分别为81.1%、95.8%、80%、59.4%，性能优于预训练版本的PaliGemma、其他VLMs以及当前最优（SotA）方法。最后，本文提出“FaceScanGPT”，这是一个基于GPT-4o的模型，可通过针对具备特定面部和/或身体属性的人物设计的提示词，识别图像中存在多个人物时的上述属性。研究结果凸显了FaceScanGPT卓越的多任务能力，仅需提示词驱动检测与识别任务，即可检测人物的发型、衣着颜色、姿态等属性。
英文摘要：
Technologies for recognizing facial attributes like race, gender, age, and emotion have several applications, such as surveillance, advertising content, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing demographic characteristics based on images and analyzing facial expressions have several challenges due to the complexity of humans' facial attributes. Traditional approaches have employed CNNs and various other deep learning techniques, trained on extensive collections of labeled images. While these methods demonstrated effective performance, there remains potential for further enhancements. In this paper, we propose to utilize vision language models (VLMs) such as generative pre-trained transformer (GPT), GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 to recognize facial attributes such as race, gender, age, and emotion from images with human faces. Various datasets like FairFace, AffectNet, and UTKFace have been utilized to evaluate the solutions. The results show that VLMs are competitive if not superior to traditional techniques. Additionally, we propose "FaceScanPaliGemma"--a fine-tuned PaliGemma model--for race, gender, age, and emotion recognition. The results show an accuracy of 81.1%, 95.8%, 80%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming pre-trained version of PaliGemma, other VLMs, and SotA methods. Finally, we propose "FaceScanGPT", which is a GPT-4o model to recognize the above attributes when several individuals are present in the image using a prompt engineered for a person with specific facial and/or physical attributes. The results underscore the superior multitasking capability of FaceScanGPT to detect the individual's attributes like hair cut, clothing color, postures, etc., using only a prompt to drive the detection and recognition tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
