MTP：面向日常对话中多模态转折点的数据集
MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations
AI总结：
本文提出将对话中的关键时刻视为转折点的新设定，构建了多模态数据集MTP，并提出框架TPMaven，利用视觉-语言模型和大语言模型检测转折点，在分类和检测任务上表现优异。
摘要：
检测对话中的关键时刻（如情绪爆发或决策改变），对于理解人类行为的转变及其后果至关重要。我们的工作引入了一个新的问题设定，将这些时刻视为转折点（TPs），并配备了一个精心构建、高共识、人工标注的多模态数据集。我们提供了精确的时间戳、描述，以及突出这些转折点在情绪、行为、观点和决策上变化的视觉-文本证据。我们还提出了一个框架TPMaven，利用最先进的视觉-语言模型从视频中构建叙事，并使用大语言模型来分类和检测我们多模态数据集中的转折点。评估结果显示，TPMaven在分类任务中达到0.88的F1分数，在检测任务中达到0.61，且补充解释与人类期望一致。
英文摘要：
Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence high-lighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
