用于音视频人员识别与验证的模态融合方法对比分析
Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification
AI总结：
该研究针对音视频人员识别与验证任务，对比三种多模态融合策略，通过实验验证了不同融合方式在两类任务上的性能差异，给出了最优方案及对应指标。
摘要：
多模态学习旨在整合来自不同模态的信息以提升学习与理解效果。本文针对人员识别与验证任务，对比了三种处理语音和人脸两种模态的融合策略。研究中，采用一维卷积神经网络从语音中提取x-vector，人脸模态则使用预训练VGGFace2网络结合迁移学习实现；此外，还将gammatonegram（伽马通语谱图）作为语音表征，搭配Darknet19预训练网络进行处理。研究使用K折交叉验证技术，在VoxCeleb2数据集测试集的118名说话人数据上对所提系统进行评估，在同等条件下对单模态系统与三种所提多模态策略开展对比评测。结果显示，gammatonegram与人脸特征的特征融合策略表现最优，在人员识别任务中准确率达98.37%；而人脸特征与x-vector拼接的方案在验证任务中的等错误率（EER）为0.62%。
英文摘要：
Multimodal learning involves integrating information from various modalities to enhance learning and comprehension. We compare three modality fusion strategies in person identification and verification by processing two modalities: voice and face. In this paper, a one-dimensional convolutional neural network is employed for x-vector extraction from voice, while the pre-trained VGGFace2 network and transfer learning are utilized for face modality. In addition, gammatonegram is used as speech representation in engagement with the Darknet19 pre-trained network. The proposed systems are evaluated using the K-fold cross-validation technique on the 118 speakers of the test set of the VoxCeleb2 dataset. The comparative evaluations are done for single-modality and three proposed multimodal strategies in equal situations. Results demonstrate that the feature fusion strategy of gammatonegram and facial features achieves the highest performance, with an accuracy of 98.37% in the person identification task. However, concatenating facial features with the x-vector reaches 0.62% for EER in verification tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
