人们的社会关系往往通过其周围环境体现出来,某些物体或互动会作为特定关系的象征,例如婚戒、玫瑰、拥抱或牵手。这给社会关系识别带来了独特挑战,需要从视觉外观中理解并捕捉这些情境的本质。然而,当前的社会关系理解方法依赖于检测到的人和物体的基础分类范式,无法理解综合情境,并且常常忽视决定性的社会因素,尤其是细微的视觉线索。为突出社会感知情境和复杂细节,我们提出一种从社会认知视角识别 Con textual So cial R elationships(ConSoR,情境社会关系)的新方法。具体而言,为融入社会感知语义,我们在冻结的 CLIP 之上构建轻量级适配器,通过新颖的多模态侧适配器调优机制学习社会概念。此外,我们为每张图像构建包含社会关系的社会感知描述性语言提示(如场景、活动、物体、情绪),随后通过视觉—语言对比促使 ConSoR 更集中地关注决定性的视觉社会因素。令人印象深刻的是,ConSoR 优于以往方法,在 People-in-Social-Context(PISC)数据集上取得 12.2% 的提升,在 People-in-Photo-Album(PIPA)基准上取得 9.8% 的提升。此外,我们观察到 ConSoR 擅长发现揭示社会关系的关键视觉证据。
英文摘要:
People's social relationships are often manifested through their surroundings, with certain objects or interactions acting as symbols for specific relationships, e.g., wedding rings, roses, hugs, or holding hands. This brings unique challenges to recognizing social relationships, requiring understanding and capturing the essence of these contexts from visual appearances. However, current methods of social relationship understanding rely on the basic classification paradigm of detected persons and objects, which fails to understand the comprehensive context and often overlooks decisive social factors, especially subtle visual cues. To highlight the social-aware context and intricate details, we propose a novel approach that recognizes \textbf{Con}textual \textbf{So}cial \textbf{R}elationships (\textbf{ConSoR}) from a social cognitive perspective. Specifically, to incorporate social-aware semantics, we build a lightweight adapter upon the frozen CLIP to learn social concepts via our novel multi-modal side adapter tuning mechanism. Further, we construct social-aware descriptive language prompts (e.g., scene, activity, objects, emotions) with social relationships for each image, and then compel ConSoR to concentrate more intensively on the decisive visual social factors via visual-linguistic contrasting. Impressively, ConSoR outperforms previous methods with a 12.2\% gain on the People-in-Social-Context (PISC) dataset and a 9.8\% increase on the People-in-Photo-Album (PIPA) benchmark. Furthermore, we observe that ConSoR excels at finding critical visual evidence to reveal social relationships.
Developing and Refining a Multifunctional Facial Recognition System for Older Adults with Cognitive Impairments: A Journey Towards Enhanced Quality of Life
Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding
Tsinghua University(清华大学);Sun Yat-sen University(中山大学);Independent Researcher(独立研究员);The Hong Kong Polytechnic University(香港理工大学);Guangdong Laboratory of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
University of Cambridge(剑桥大学);Shenzhen University(深圳大学);Shenzhen Institute of Artificial Intelligence of Robotics of Society(深圳人工智能与机器人学会);University of Nottingham(诺丁汉大学)
Academy for Engineering & Technology, Fudan University(复旦大学工程与技术学院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Yiwu Research Institute, Fudan University(复旦大学义乌研究院)