从自然对齐到条件可控性在多模态对话中
From Natural Alignment to Conditional Controllability in Multimodal Dialogue
AI总结:
本文提出MM-Dia数据集和MM-Dia-Bench测试集,通过多模态条件控制提升对话生成的可控性和表达性,实验表明现有框架难以复现人类交互的细腻表达。
摘要:
最近的人工智能生成内容(AIGC)的进步在建模人类交互,特别是在多模态对话领域取得了显著进展。尽管现有方法在孤立模态如语音或视觉中能生成逼真的对话,但可控多模态对话生成(MDG)仍面临挑战。本文聚焦人类交互中语音、视觉和文本的自然对齐,旨在通过多模态条件控制实现富有表现力的对话生成。为解决现有数据集在对话表达性上的不足,我们引入了一种新的多模态对话注释流程,从电影和电视剧中精选对话并进行细粒度注释。所得到的MM-Dia数据集(360多小时,54,700条对话)促进了显式的MDG控制,特别是通过风格可控的对话语音合成。同时,MM-Dia-Bench(309条高表达力对话,包含可见的单/双说话人场景)作为严格测试平台,评估隐式的跨模态MDG控制,检查音频-视觉风格一致性。大量实验表明,训练MM-Dia显著提升了细粒度可控性,而MM-Dia-Bench的评估揭示了现有框架在复制人类交互细腻表达方面的局限性。这些发现为多模态条件对话生成提供了新的见解和挑战。
英文摘要:
The recent advancement of Artificial Intelligence Generated Content (AIGC) has led to significant strides in modeling human interaction, particularly in the context of multimodal dialogue. While current methods impressively generate realistic dialogue in isolated modalities like speech or vision, challenges remain in controllable Multimodal Dialogue Generation (MDG). This paper focuses on the natural alignment between speech, vision, and text in human interaction, aiming for expressive dialogue generation through multimodal conditional control. To address the insufficient richness and diversity of dialogue expressiveness in existing datasets, we introduce a novel multimodal dialogue annotation pipeline to curate dialogues from movies and TV series with fine-grained annotations in interactional characteristics. The resulting MM-Dia dataset (360+ hours, 54,700 dialogues) facilitates explicitly controlled MDG, specifically through style-controllable dialogue speech synthesis. In parallel, MM-Dia-Bench (309 highly expressive dialogues with visible single-/dual-speaker scenes) serves as a rigorous testbed for implicit cross-modal MDG control, evaluating audio-visual style consistency across modalities. Extensive experiments demonstrate that training on MM-Dia significantly enhances fine-grained controllability, while evaluations on MM-Dia-Bench reveal limitations in current frameworks to replicate the nuanced expressiveness of human interaction. These findings provides new insights and challenges for multimodal conditional dialogue generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。