arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.08358cs.CVcs.AI
收藏

从社会认知视角出发:情境感知的视觉社会关系识别

From a Social Cognitive Perspective: Context-aware Visual Social Relationship Recognition

University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)

Shiwei Wu, Chao Zhang, Joya Chen, Tong Xu, Likang Wu, Yao Hu, Enhong Chen

AI总结:

本文从社会认知视角提出 ConSoR,通过冻结 CLIP 上的多模态侧适配器调优和视觉—语言对比学习社会情境,在 PISC 与 PIPA 上分别提升 12.2% 和 9.8%。

摘要:

人们的社会关系往往通过其周围环境体现出来,某些物体或互动会作为特定关系的象征,例如婚戒、玫瑰、拥抱或牵手。这给社会关系识别带来了独特挑战,需要从视觉外观中理解并捕捉这些情境的本质。然而,当前的社会关系理解方法依赖于检测到的人和物体的基础分类范式,无法理解综合情境,并且常常忽视决定性的社会因素,尤其是细微的视觉线索。为突出社会感知情境和复杂细节,我们提出一种从社会认知视角识别 Con textual So cial R elationships(ConSoR,情境社会关系)的新方法。具体而言,为融入社会感知语义,我们在冻结的 CLIP 之上构建轻量级适配器,通过新颖的多模态侧适配器调优机制学习社会概念。此外,我们为每张图像构建包含社会关系的社会感知描述性语言提示(如场景、活动、物体、情绪),随后通过视觉—语言对比促使 ConSoR 更集中地关注决定性的视觉社会因素。令人印象深刻的是,ConSoR 优于以往方法,在 People-in-Social-Context(PISC)数据集上取得 12.2% 的提升,在 People-in-Photo-Album(PIPA)基准上取得 9.8% 的提升。此外,我们观察到 ConSoR 擅长发现揭示社会关系的关键视觉证据。

英文摘要:

People's social relationships are often manifested through their surroundings, with certain objects or interactions acting as symbols for specific relationships, e.g., wedding rings, roses, hugs, or holding hands. This brings unique challenges to recognizing social relationships, requiring understanding and capturing the essence of these contexts from visual appearances. However, current methods of social relationship understanding rely on the basic classification paradigm of detected persons and objects, which fails to understand the comprehensive context and often overlooks decisive social factors, especially subtle visual cues. To highlight the social-aware context and intricate details, we propose a novel approach that recognizes \textbf{Con}textual \textbf{So}cial \textbf{R}elationships (\textbf{ConSoR}) from a social cognitive perspective. Specifically, to incorporate social-aware semantics, we build a lightweight adapter upon the frozen CLIP to learn social concepts via our novel multi-modal side adapter tuning mechanism. Further, we construct social-aware descriptive language prompts (e.g., scene, activity, objects, emotions) with social relationships for each image, and then compel ConSoR to concentrate more intensively on the decisive visual social factors via visual-linguistic contrasting. Impressively, ConSoR outperforms previous methods with a 12.2\% gain on the People-in-Social-Context (PISC) dataset and a 9.8\% increase on the People-in-Photo-Album (PIPA) benchmark. Furthermore, we observe that ConSoR excels at finding critical visual evidence to reveal social relationships.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑