arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.04366cs.HCcs.MM
收藏

向人类专家对齐多模态大语言模型:聚焦亲子互动

Towards Aligning Multimodal LLMs with Human Experts: A Focus on Parent-Child Interaction

Singapore University of Technology and Design(新加坡科技设计大学)

Weiyan Shi, Kenny Tsu Wei Choo

AI总结:

本文探讨了如何通过两阶段提示方法使多模态大语言模型更有效地对齐语言治疗师在分析亲子互动中的联合注意,揭示了观察层与判断层对齐的差异及挑战。

摘要:

尽管多模态大语言模型(MLLMs)越来越多地应用于以人为本的人工智能系统,但其理解复杂社会互动的能力仍存疑。我们对齐MLLMs与语言治疗师(SLPs)在分析亲子互动中的联合注意,这是早期社会沟通发展中的关键构念。基于与三位SLPs的访谈和视频注释,我们描述了目光、动作和发声的观察线索如何影响他们的推理过程。然后我们测试了MLLM能否通过两阶段提示方法近似这一流程,将观察与判断分开。我们的发现表明,对齐在观察层更稳健,因为专家共享共同的描述,而在判断层,解释标准分歧。我们将这项工作定位为一种基于案例的探针,探讨复杂社会行为中的专家-人工智能对齐,突显了将MLLMs应用于社会情境互动分析的可行性和挑战。

英文摘要:

While multimodal large language models (MLLMs) are increasingly applied in human-centred AI systems, their ability to understand complex social interactions remains uncertain. We present an exploratory study on aligning MLLMs with speech-language pathologists (SLPs) in analysing joint attention in parent-child interactions, a key construct in early social-communicative development. Drawing on interviews and video annotations with three SLPs, we characterise how observational cues of gaze, action, and vocalisation inform their reasoning processes. We then test whether an MLLM can approximate this workflow through a two-stage prompting approach, separating observation from judgement. Our findings reveal that alignment is more robust at the observation layer, where experts share common descriptors, than at the judgement layer, where interpretive criteria diverge. We position this work as a case-based probe into expert-AI alignment in complex social behaviour, highlighting both the feasibility and the challenges of applying MLLMs to socially situated interaction analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑