arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.09255cs.CV

FVQ:面向人脸视频质量评估的大规模数据集与基于大多模态模型的方法

FVQ: A Large-Scale Dataset and an LMM-based Method for Face Video Quality Assessment

Shanghai Jiao Tong University(上海交通大学);East China Normal University(华东师范大学)

Sijing Wu, Yunhao Li, Ziwen Xu, Yixuan Gao, Huiyu Duan, Wei Sun, Guangtao Zhai

AI总结:

针对人脸视频质量评估缺乏大规模数据集的问题,该研究构建了含2万条视频的FVQ-20K数据集,提出基于大多模态模型的FVQ-Rater方法,在相关数据集上取得优异表现,助力领域发展。

摘要:

除通用视频质量评估(VQA)外,人脸视频质量评估(FVQA)值得深入探究,因为人脸视频是社交媒体平台的核心内容,且人类视觉系统(HVS)对人脸格外敏感。但由于缺乏大规模FVQA数据集,该领域目前研究较少。为填补这一空白,我们构建了首个大规模真实场景FVQA数据集FVQ-20K,包含20000条真实场景人脸视频及对应的平均意见得分(MOS)标注。在FVQ-20K数据集的基础上,我们进一步提出了专用FVQA方法FVQ-Rater,实现人脸视频的类人评级与打分,这是首次探索大多模态模型(LMMs)在FVQA任务中的潜力。具体而言,我们精心提取了空间特征、时间特征以及人脸专属特征(即人像特征和人脸嵌入)等多维度特征,以提供全面的视觉信息,并利用基于LoRA的指令微调技术实现面向质量的专项微调,该方法在FVQ-20K和CFVQA数据集上均表现出优异性能。大量实验与综合分析表明,FVQ-20K数据集和FVQ-Rater方法对推动FVQA领域发展具有显著潜力。

英文摘要:

Face video quality assessment (FVQA) deserves to be explored in addition to general video quality assessment (VQA), as face videos are the primary content on social media platforms and human visual system (HVS) is particularly sensitive to human faces. However, FVQA is rarely explored due to the lack of large-scale FVQA datasets. To fill this gap, we present the first large-scale in-the-wild FVQA dataset, FVQ-20K, which contains 20,000 in-the-wild face videos together with corresponding mean opinion score (MOS) annotations. Along with the FVQ-20K dataset, we further propose a specialized FVQA method named FVQ-Rater to achieve human-like rating and scoring for face video, which is the first attempt to explore the potential of large multimodal models (LMMs) for the FVQA task. Concretely, we elaborately extract multi-dimensional features including spatial features, temporal features, and face-specific features (i.e., portrait features and face embeddings) to provide comprehensive visual information, and take advantage of the LoRA-based instruction tuning technique to achieve quality-specific fine-tuning, which shows superior performance on both FVQ-20K and CFVQA datasets. Extensive experiments and comprehensive analysis demonstrate the significant potential of the FVQ-20K dataset and FVQ-Rater method in promoting the development of FVQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑