arXiv 2505.202772025-06-05cs.CLcs.CV

OmniCharacter：面向沉浸式角色扮演智能体的无缝语音-语言人格交互

OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

Tongji University（同济大学）；Tongyi Laboratory（通义实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）

Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting-En Lin, Pengpeng Zeng, Qiang Qu, Feiteng Fang, Min Yang, Lianli Gao, Jingkuan Song, Fei Huang, Yongbin Li

备注：: 14 pages, 6 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

OmniCharacter 提出首个无缝语音-语言人格交互模型，通过融合角色人格与声音特征并构建 OmniCharacter-10K 数据集，在低至 289ms 延迟下实现更沉浸、内容与风格更优的角色扮演智能体。

摘要：

角色扮演智能体（Role-Playing Agents, RPAs）受益于大语言模型，是一种新兴的交互式人工智能系统，能够模拟具有多样化人格的角色或人物。然而，现有方法主要聚焦于以文本形式模仿角色之间的对话，忽视了角色的声音特征（如语音风格和情感）在交互中所起的关键作用，而这些特征往往能在真实场景中带来更强的沉浸式体验。为了实现这一目标，我们提出了 OmniCharacter，这是首个无缝语音-语言人格交互模型，旨在以低延迟实现沉浸式角色扮演智能体。具体而言，OmniCharacter 使智能体能够在整个交互过程中一致地展现角色特定的人格特征和声音特征，从而支持语音与语言混合响应。为了使模型与语音-语言场景对齐，我们构建了一个名为 OmniCharacter-10K 的数据集，该数据集包含更具区分度的角色（20 个）、丰富情境化的多轮对话（10K）以及动态语音响应（135K）。实验结果表明，与现有角色扮演智能体和主流语音-语言模型相比，我们的方法在内容和风格方面均能产生更优的响应，响应延迟低至 289 毫秒。代码和数据集可在 https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/OmniCharacter 获取。

英文摘要：

Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role's voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms. Code and dataset are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/OmniCharacter.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。