arXiv 2603.169662026-03-17cs.CVcs.AIcs.MMcs.SDeess.AS

CineSRD：利用视觉、听觉和语言线索进行开放世界视觉媒体说话人聚类

CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization

Hujing Digital Media and Entertainment Group（华景数字媒体与娱乐集团）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部门）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）

Liangbin Huang, Xiaohua Liao, Chaoqun Cui, Shijing Wang, Zhaolong Huang, Yanlong Du, Wenji Mao

备注：: Accepted to CVPR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

CineSRD通过整合视觉、听觉和语言线索，解决开放世界视觉媒体中说话人识别的挑战，提出统一多模态框架并构建专用基准数据集，验证其在复杂场景下的鲁棒性和泛化能力。

摘要：

传统说话人聚类系统主要针对受限场景如会议和采访，其中说话人数量有限且声音条件较为清晰。为探索开放世界说话人聚类，我们扩展此任务至视觉媒体领域，涵盖复杂音频视觉节目如电影和电视剧。此新场景引入了多个挑战，包括长视频理解、大量说话人、音频与视觉线索的跨模态异步性以及不受控的现实场景变化。为解决这些挑战，我们提出了Cinematic Speaker Registration & Diarization (CineSRD)，一个统一的多模态框架，利用视频、语音和字幕中的视觉、听觉和语言线索进行说话人标注。CineSRD首先执行视觉锚点聚类以注册初始说话人，然后整合音频语言模型进行说话人轮次检测，优化标注并补充未注册的非屏幕说话人。此外，我们构建并发布了一个专用的说话人聚类基准数据集，包含中英文节目。实验结果表明，CineSRD在所提出基准上取得优越性能，并在传统数据集上取得竞争性结果，验证了其在开放世界视觉媒体设置中的鲁棒性和泛化能力。

英文摘要：

Traditional speaker diarization systems have primarily focused on constrained scenarios such as meetings and interviews, where the number of speakers is limited and acoustic conditions are relatively clean. To explore open-world speaker diarization, we extend this task to the visual media domain, encompassing complex audiovisual programs such as films and TV series. This new setting introduces several challenges, including long-form video understanding, a large number of speakers, cross-modal asynchrony between audio and visual cues, and uncontrolled in-the-wild variability. To address these challenges, we propose Cinematic Speaker Registration & Diarization (CineSRD), a unified multimodal framework that leverages visual, acoustic, and linguistic cues from video, speech, and subtitles for speaker annotation. CineSRD first performs visual anchor clustering to register initial speakers and then integrates an audio language model for speaker turn detection, refining annotations and supplementing unregistered off-screen speakers. Furthermore, we construct and release a dedicated speaker diarization benchmark for visual media that includes Chinese and English programs. Experimental results demonstrate that CineSRD achieves superior performance on the proposed benchmark and competitive results on conventional datasets, validating its robustness and generalizability in open-world visual media settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。