利用多模态语言模型评估婴儿的视觉和语言经验一致性
Assessing the alignment between infants' visual and linguistic experience using multimodal language models
AI总结：
基于多模态语言模型评估婴儿视觉与语言经验一致性，揭示日常学习中视觉与语言对齐的稀有性及跨儿童差异。
摘要：
确定词语所指对象是幼儿语言学习的核心挑战。大多数模型认为儿童通过他人谈论物体时词与指代物的共现学习早期物体标签。但日常学习中儿童的视觉与语言经验在时间上有多同步吗？迄今为止，这一问题的答案受限于需要大量手动注释视觉-语言共现。本文评估了对比语言-图像预训练（CLIP）模型自动表征婴儿视角的日常视频中视觉-语言对齐情况。在验证CLIP对齐评分后，我们将其应用于大量婴儿视角视频。我们发现，理想对齐时刻（如“看球”时球在儿童视野中）在儿童日常经验中比现代机器学习数据集稀少，且在儿童内部和跨儿童之间均存在差异。这些发现表明，稀少的对齐是描述早期词汇学习模型的限制，并提供了一种研究儿童多模态环境的新方法。
英文摘要：
Figuring out which objects or concepts words refer to is a central language learning challenge for young children. Most models of this process posit that children learn early object labels from co-occurrences of words and their referents that occur when someone around them talks about an object in the immediate physical environment. But how aligned in time are children's visual and linguistic experiences during everyday learning? To date, answers to this question have been limited by the need for labor-intensive manual annotations of vision-language co-occurrences. Here, we evaluate the use of contrastive language-image pretraining (CLIP) models to automatically characterize vision-language alignment in egocentric videos taken from the infant perspective in home environments. After validating CLIP alignment scores using human alignment judgments, we apply this metric to a large corpus of infant-perspective videos. We show that idealized aligned moments for learning (e.g., "look at the ball" with a ball present in the child's view) are relatively rare in children's everyday experiences compared to modern machine learning datasets, and highlight variability in alignment both within and across children. These findings suggest that infrequent alignment is a constraint for models describing early word learning and offer a new method for investigating children's multimodal environment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
