arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.23947cs.HC
收藏

大语言模型的社会注视:人类行为理解的多模态方法文献综述

The Social Gaze of LLMs: A Literature Review of Multimodal Approaches to Human Behavior Understanding

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Zihan Liu, Parisa Rabbani, Veda Duddu, Kyle Fan, Madison Lee, Yun Huang

AI总结:

本文针对176篇相关文献开展系统性综述,借助四维编码框架分析LLM多模态系统在人类行为理解应用中的技术、评估与伦理现存问题,提出对应研究议程。

摘要:

由大语言模型(LLM)驱动的多模态系统正越来越多地被用于解读人类行为,但研究人员如何应用这些模型的“社会能力”仍缺乏清晰认知。本文对医疗、教育、娱乐等不同应用领域的176篇出版物开展了系统性文献综述。借助包含应用、技术、评估、伦理四个维度的编码框架,我们发现四点结论:一是现有研究频繁使用多模态源的模式识别与信息提取技术,但对自适应交互式推理的支持有限;二是主流采用“模态转文本”流水线,更侧重语言而弱化丰富的视听线索,丢失了细微的社会线索;三是评估实践依赖静态基准,基于社会场景、以人为本的评估十分少见;四是伦理讨论主要聚焦隐私等法律与权利相关风险,欺骗等社会风险被忽视,最多仅被提及但未得到解决。我们提出了一套评估具备社会能力、符合伦理要求、具备交互感知能力的多模态系统的研究议程。

英文摘要:

LLM-powered multimodal systems are increasingly used to interpret human behavior, yet how researchers apply the models' 'social competence' remains poorly understood. This paper presents a systematic literature review of 176 publications across different application domains (e.g., healthcare, education, and entertainment). Using a four-dimensional coding framework (application, technical, evaluative, and ethical), we find (1) frequent use of pattern recognition and information extraction from multimodal sources, but limited support for adaptive, interactive reasoning; (2) a dominant 'modality-to-text' pipeline that privileges language over rich audiovisual cues, striping away nuanced social cues; (3) evaluation practices reliant on static benchmarks, with socially grounded, human-centered assessments rare; and (4) Ethical discussions focused mainly on legal and rights-related risks (e.g., privacy), leaving societal risks (e.g., deception) overlooked--or at best acknowledged but left unaddressed. We outline a research agenda for evaluating socially competent, ethically informed, and interaction-aware multi-modal systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑