使用大型多模态模型在面部图像中进行零样本情感标注：多类别、多帧方法的基准测试与前景
Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches
AI总结：
本研究探讨了使用GPT-4o-mini模型在FERV39k数据集上进行零样本面部情感标注的可行性，发现三元分类准确率（约64%）优于七类分类（约50%），且多帧整合策略可略微提升标注准确率并降低成本。
摘要：
本研究调查了在日常生活场景中使用大型多模态模型（LMMs）自动标注人类情感的可行性与性能。我们在公开的 FERV39k 数据集的 DailyLife 子集上进行了实验，采用 GPT-4o-mini 模型对从视频片段中提取的关键帧进行快速零样本标注。在七类情感分类法（“愤怒”、“厌恶”、“恐惧”、“快乐”、“中性”、“悲伤”、“惊讶”）下，LMM 的平均准确率约为 50%。相比之下，当限制为三元情感分类（负面/中性/正面）时，平均准确率提升至约 64%。此外，我们探索了一种整合 1-2 秒视频片段内多帧的策略，以提升标注性能并降低成本。结果表明，该方法能略微提高标注准确率。总体而言，我们的初步发现突出了零样本 LMMs 在人类面部情感标注任务中的潜在应用，为降低标注成本和拓展 LMMs 在复杂多模态环境中的适用性提供了新途径。
英文摘要：
This study investigates the feasibility and performance of using large multimodal models (LMMs) to automatically annotate human emotions in everyday scenarios. We conducted experiments on the DailyLife subset of the publicly available FERV39k dataset, employing the GPT-4o-mini model for rapid, zero-shot labeling of key frames extracted from video segments. Under a seven-class emotion taxonomy ("Angry," "Disgust," "Fear," "Happy," "Neutral," "Sad," "Surprise"), the LMM achieved an average precision of approximately 50%. In contrast, when limited to ternary emotion classification (negative/neutral/positive), the average precision increased to approximately 64%. Additionally, we explored a strategy that integrates multiple frames within 1-2 second video clips to enhance labeling performance and reduce costs. The results indicate that this approach can slightly improve annotation accuracy. Overall, our preliminary findings highlight the potential application of zero-shot LMMs in human facial emotion annotation tasks, offering new avenues for reducing labeling costs and broadening the applicability of LMMs in complex multimodal environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
