EgoSocial：通过自我中心社会交互感知基准测试全模态大语言模型的主动干预能力
EgoSocial: Benchmarking Proactive Intervention Ability of Omnimodal LLMs via Egocentric Social Interaction Perception
AI总结：
针对LLM缺乏社交干预意识的问题，提出EgoSocial基准数据集和EgoSoD方法，通过多模态社交思维图动态建模交互，显著提升干预时机检测性能。
摘要：
随着增强现实（AR）和虚拟现实（VR）技术日益融入日常生活，人们对能够从自我中心视角理解人类社交动态的人工智能的需求不断增长。然而，当前的大语言模型（LLMs）往往缺乏作为AI助手判断何时进行干预所需的社交意识。这导致模型持续产生缺乏社交感知的回应，可能扰乱自然对话并对用户专注度产生负面影响。为解决这些局限，我们引入了EgoSocial，一个包含13,500个社交视频-问题对的大规模自我中心数据集，专门用于基准测试社交交互感知中的干预能力。我们还对当前全模态大语言模型（OLLMs）进行了深入分析，以评估其在检测多样化社交情境线索方面的有效性。实验表明，OLLMs在检测干预时机方面仍存在困难（Gemini 2.5 Pro的准确率为14.4%）。我们还提出了EgoSoD（EgoSocial检测），一种用于稳健识别社交动态的端到端方法。基于我们的OLLM分析，EgoSoD将多模态情境线索（如音频和视觉线索）整合到社交思维图中，动态建模参与者和交互。我们的方法主动检测干预时机和社交交互，精确确定何时进行干预。我们的EgoSoD在干预时机性能上使Phi-4提升45.6%，使Gemini 2.5 Pro提升9.9%，在整体社交交互性能上使Phi-4提升20.4%，使Gemini 2.5 Pro提升6.9%。我们将很快发布数据集和代码。
英文摘要：
As AR/VR technologies become integral to daily life, there's a growing need for AI that understands human social dynamics from an egocentric perspective. However, current LLMs often lack the social awareness to discern when to intervene as AI assistant. This leads to constant, socially unaware responses that may disrupt natural conversation and negatively impact user focus. To address these limitations, we introduce EgoSocial, a large-scale egocentric dataset with 13,500 social video-question pairs, specifically designed to benchmark intervention in social interaction perception. We also present an in-depth analysis of current omnimodal LLMs (OLLMs) to assess their effectiveness in detecting diverse social contextual cues. Experiments show that OLLMs still struggle to detect the intervention timing (14.4% for Gemini 2.5 Pro). We also propose EgoSoD (EgoSocial Detection), an end-to-end method for robustly discerning social dynamics. Informed by our OLLM analysis, EgoSoD integrates multimodal contextual cues (e.g., audio and visual cues) into a social thinking graph, dynamically modeling participants and interactions. Our method proactively detects intervention timing and social interactions, precisely determining when to intervene. Our EgoSoD improves Phi-4 by 45.6% and Gemini 2.5 Pro by 9.9% on Intervention Timing performance, and improves Phi-4 by 20.4% and Gemini 2.5 Pro by 6.9% on overall Social Interaction performance. We will release the dataset and code soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
