Video2Roleplay：面向视频引导角色扮演智能体的多模态数据集与框架
Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents
AI总结：
本文提出Video2Roleplay框架，通过引入视频模态构建动态角色画像，并发布含6万视频和70万对话的Role-playing-Video60k数据集，结合自适应时间采样与静态画像，显著提升角色扮演智能体的响应质量。
摘要：
角色扮演智能体（RPA）因其模拟沉浸式交互角色的能力而受到越来越多的关注。然而，现有方法主要聚焦于静态角色画像，忽视了人类固有的动态感知能力。为弥合这一差距，我们通过将视频模态引入RPA，提出了动态角色画像的概念。为支持这一工作，我们构建了Role-playing-Video60k，一个包含6万个视频和70万条对应对话的大规模高质量数据集。基于该数据集，我们开发了一个综合性的RPA框架，该框架将自适应时间采样与动态和静态角色画像表示相结合。具体而言，动态画像通过自适应采样视频帧并按时间顺序输入大语言模型（LLM）来创建，而静态画像由（1）微调期间来自训练视频的角色对话，以及（2）推理期间来自输入视频的摘要上下文组成。这种联合集成使RPA能够生成更优的响应。此外，我们提出了一种涵盖八项指标的稳健评估方法。实验结果证明了我们框架的有效性，凸显了动态角色画像在开发RPA中的重要性。
英文摘要：
Role-playing agents (RPAs) have attracted growing interest for their ability to simulate immersive and interactive characters. However, existing approaches primarily focus on static role profiles, overlooking the dynamic perceptual abilities inherent to humans. To bridge this gap, we introduce the concept of dynamic role profiles by incorporating video modality into RPAs. To support this, we construct Role-playing-Video60k, a large-scale, high-quality dataset comprising 60k videos and 700k corresponding dialogues. Based on this dataset, we develop a comprehensive RPA framework that combines adaptive temporal sampling with both dynamic and static role profile representations. Specifically, the dynamic profile is created by adaptively sampling video frames and feeding them to the LLM in temporal order, while the static profile consists of (1) character dialogues from training videos during fine-tuning, and (2) a summary context from the input video during inference. This joint integration enables RPAs to generate greater responses. Furthermore, we propose a robust evaluation method covering eight metrics. Experimental results demonstrate the effectiveness of our framework, highlighting the importance of dynamic role profiles in developing RPAs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
