arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08914cs.HCcs.AI
收藏

指向抽象:协作物理任务中的多模态惯例形成

Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks

Princeton University(普林斯顿大学);Brown University(布朗大学);MIT(麻省理工学院);Stanford University(斯坦福大学)

Kiyosu Maeda, William P. McCarthy, Ching-Yi Tsai, Jeffrey Mu, Haoliang Wang, Robert D. Hawkins, Judith E. Fan, Parastoo Abtahi

AI总结:

研究通过多模态协作任务探讨了如何通过建立语言和手势抽象来提高协作效率,并扩展了惯例形成的概率模型。

摘要:

人类智能的一个本质特征是能够随时间创造临时惯例以高效实现共享目标。我们研究了通过重复协作如何演变通信策略,以协调共享的程序抽象。为此,我们进行了一项在线单模研究(n=98),使用自然语言探测抽象层次。在后续的实验室研究(n=40)中,我们考察了在物理协作过程中多模态通信(语音和手势)如何变化。参与者使用增强现实技术隔离对方的手和声音;一人观看3D虚拟塔楼并发送指令给另一人,后者构建实物塔楼。参与者通过建立语言和手势抽象,并利用跨模态冗余强调关键变化来提高速度和准确性。基于这些发现,我们扩展了惯例形成的概率模型到多模态环境,捕捉模态偏好的变化。我们的发现和模型为设计在物理世界中具备惯例意识的智能体提供了构建块。

英文摘要:

A quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner's hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑