社会基因组：多模态模型的基于证据的社会推理能力
Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
AI总结：
本文提出首个细粒度、基于证据的多模态社会推理基准SOCIAL GENOME，包含272个视频和1,486条人工标注推理轨迹，并通过实验揭示现有多模态模型的性能差距。
摘要：
社会推理能力对于人工智能系统在社会情境中有效解读和回应多模态人类交流与互动至关重要。我们提出SOCIAL GENOME，这是首个用于细粒度、基于证据的多模态模型社会推理能力基准。SOCIAL GENOME包含272个互动视频和1,486条与这些互动推断相关的人工标注推理轨迹。这些轨迹包含5,777个推理步骤，引用了来自视觉线索、言语线索、声音线索以及外部知识（视频之外的情境知识）的证据。SOCIAL GENOME也是首个研究社会推理中外部知识的建模挑战。SOCIAL GENOME计算指标以整体评估模型生成的社会推理轨迹的语义和结构质量。我们通过与最先进模型的实验展示了SOCIAL GENOME的实用性，识别了性能差距以及未来研究改进多模态模型基于证据的社会推理能力的机会。
英文摘要：
Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. SOCIAL GENOME contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). SOCIAL GENOME is also the first modeling challenge to study external knowledge in social reasoning. SOCIAL GENOME computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of SOCIAL GENOME through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
