arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.19792cs.CLcs.AIcs.HC
收藏

LVLMs与人类在指称交流中的基础不同

LVLMs and Humans Ground Differently in Referential Communication

Department of Computer Science(计算机科学系);Department of Psychology(心理学系);Department of Linguistics(语言学系);Institute for Advanced Computational Science(先进计算科学研究院)

Peter Zeng, Weiling Li, Amie J. Paige, Zhengxiang Wang, Panagiotis Kaliosis, Dimitris Samaras, Gregory Zelinsky, Susan E. Brennan, Owen Rambow

AI总结:

通过人类与AI配对的多轮指称交流实验,发现LVLMs无法像人类一样利用共同基础生成和解析指称表达,导致交流不畅。

摘要:

对于生成式AI代理与人类用户有效合作,准确预测人类意图的能力至关重要。但这种协作能力仍然受到一个关键缺陷的限制:无法建模共同基础。我们提出了一个因子设计的指称交流实验,涉及指导者-匹配者配对(人类-人类、人类-AI、AI-人类和AI-AI),他们在多轮重复回合中交互,以匹配与任何明显词汇化标签无关的物体图片。我们表明,LVLMs无法以促进顺畅交流的方式交互式生成和解析指称表达,而这是人类语言使用的基础技能。我们发布了包含356个对话(89对,每对4轮)的语料库,以及用于数据收集的在线流程和用于分析准确性、效率和词汇重叠的工具。

英文摘要:

For generative AI agents to partner effectively with human users, the ability to accurately predict human intent is critical. But this ability to collaborate remains limited by a critical deficit: an inability to model common ground. We present a referential communication experiment with a factorial design involving director-matcher pairs (human-human, human-AI, AI-human, and AI-AI) that interact with multiple turns in repeated rounds to match pictures of objects not associated with any obvious lexicalized labels. We show that LVLMs cannot interactively generate and resolve referring expressions in a way that enables smooth communication, a crucial skill that underlies human language use. We release our corpus of 356 dialogues (89 pairs over 4 rounds each) along with the online pipeline for data collection and the tools for analyzing accuracy, efficiency, and lexical overlap.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑