超越语言的表达：一个用于从视频对话中学习非语言线索的大规模多模态数据集
Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues
AI总结：
本文提出多模态语言模型MARS及大规模视频对话数据集VENUS，通过将文本与向量量化的非语言表示统一训练，使模型能够同时理解并生成语言与非语言线索。
摘要：
非语言交流是人类互动中不可或缺的一部分，手势、面部表情和肢体语言传递着意图和情感的关键信息。然而，现有的大语言模型（LLMs）未能有效整合这些非语言元素，限制了它们创造完全沉浸式对话体验的能力。我们提出了MARS，一个多模态语言模型，旨在理解并生成伴随文本的非语言线索，从而弥合对话人工智能中的这一差距。我们的关键创新是VENUS，一个大规模数据集，包含带时间对齐文本、面部表情和肢体语言标注的视频。利用VENUS，我们以下一标记预测为目标训练MARS，将文本与向量量化的非语言表示相结合，在统一框架内实现多模态理解与生成。基于对VENUS数据集的多项分析，我们验证了其庞大的规模和高效性。我们的定量和定性结果表明，MARS能够根据对话输入成功生成文本和非语言表达。
英文摘要：
Nonverbal communication is integral to human interaction, with gestures, facial expressions, and body language conveying critical aspects of intent and emotion. However, existing large language models (LLMs) fail to effectively incorporate these nonverbal elements, limiting their capacity to create fully immersive conversational experiences. We introduce MARS, a multimodal language model designed to understand and generate nonverbal cues alongside text, bridging this gap in conversational AI. Our key innovation is VENUS, a large-scale dataset comprising annotated videos with time-aligned text, facial expressions, and body language. Leveraging VENUS, we train MARS with a next-token prediction objective, combining text with vector-quantized nonverbal representations to achieve multimodal understanding and generation within a unified framework. Based on various analyses of the VENUS datasets, we validate its substantial scale and high effectiveness. Our quantitative and qualitative results demonstrate that MARS successfully generates text and nonverbal languages, corresponding to conversational input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
