MIKU-PAL：一种用于语音副语言与情感标注的自动化、标准化多模态方法
MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling
AI总结：
本文提出自动化多模态流水线 MIKU-PAL，结合人脸检测跟踪与 MLLM 从未标注视频中提取高一致性情感语音，并发布 131.2 小时细粒度数据集 MIKU-EmoBench。
摘要：
获取具有强一致性的大规模情感语音数据，对于语音合成而言仍是一项挑战。本文提出 MIKU-PAL，这是一条全自动化的多模态流水线，用于从未标注视频数据中提取高一致性的情感语音。借助人脸检测与跟踪算法，我们利用多模态大语言模型（MLLM）开发了一套自动情感分析系统。结果表明，MIKU-PAL 能够达到人类水平的准确率（在 MELD 上为 68.5%）和更优的一致性（Fleiss kappa 得分为 0.93），同时比人工标注成本更低、速度更快。凭借 MIKU-PAL 提供的高质量、灵活且一致的标注，我们能够标注多达 26 种细粒度语音情感类别，并由人工标注者验证，获得 83% 的合理性评分。基于所提出的系统，我们进一步发布了细粒度情感语音数据集 MIKU-EmoBench（131.2 小时），作为情感文本转语音和视觉声音克隆的新基准。
英文摘要：
Acquiring large-scale emotional speech data with strong consistency remains a challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated multimodal pipeline for extracting high-consistency emotional speech from unlabeled video data. Leveraging face detection and tracking algorithms, we developed an automatic emotion analysis system using a multimodal large language model (MLLM). Our results demonstrate that MIKU-PAL can achieve human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss kappa score) while being much cheaper and faster than human annotation. With the high-quality, flexible, and consistent annotation from MIKU-PAL, we can annotate fine-grained speech emotion categories of up to 26 types, validated by human annotators with 83% rationality ratings. Based on our proposed system, we further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2 hours) as a new benchmark for emotional text-to-speech and visual voice cloning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
