arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.22948cs.AI
收藏

语言、视觉和动作表征的一致性

Alignment among Language, Vision and Action Representations

Institute of Cognitive Sciences and Technologies, National Research Council(国家研究委员会认知科学与技术研究所);University of Naples “Federico II”(那不勒斯费德里科二世大学)

Nicola Milano, Stefano Nolfi

AI总结:

研究通过训练Transformer智能体执行自然语言指令,发现语言、视觉和动作表征在跨模态中呈现部分共享的语义结构,支持模态无关的语义组织。

摘要:

认知科学和人工智能领域的一个基本问题在于,不同的学习模态:语言、视觉和动作,是否会产生不同的或共享的内部表征。传统观点认为,训练于不同数据类型的模型会发展出专门的、不可转移的表征。然而,最近的证据表明出现了意想不到的收敛:为不同任务优化的模型可能发展出相似的表征几何结构。我们研究这种收敛是否扩展到具身动作学习,通过训练一个基于Transformer的智能体来响应自然语言指令执行目标导向的行为。使用BabyAI平台上的行为克隆,我们生成了由传感器运动控制需求塑造的行动接地语言嵌入。然后我们将这些表征与提取自最新大型语言模型(LLaMA、Qwen、DeepSeek、BERT)和视觉-语言模型(CLIP、BLIP)的表征进行比较。尽管训练数据、模态和目标存在显著差异,我们观察到稳健的跨模态对齐。动作表征与解码器-only语言模型和BLIP(精度@15:0.70-0.73)的对齐程度较强,接近语言模型之间的对齐程度。与CLIP和BERT的对齐程度显著较弱。这些发现表明,语言、视觉和动作表征趋于部分共享的语义结构,支持模态无关的语义组织,并强调了在具身AI系统中跨领域转移的潜力。

英文摘要:

A fundamental question in cognitive science and AI concerns whether different learning modalities: language, vision, and action, give rise to distinct or shared internal representations. Traditional views assume that models trained on different data types develop specialized, non-transferable representations. However, recent evidence suggests unexpected convergence: models optimized for distinct tasks may develop similar representational geometries. We investigate whether this convergence extends to embodied action learning by training a transformer-based agent to execute goal-directed behaviors in response to natural language instructions. Using behavioral cloning on the BabyAI platform, we generated action-grounded language embeddings shaped exclusively by sensorimotor control requirements. We then compared these representations with those extracted from state-of-the-art large language models (LLaMA, Qwen, DeepSeek, BERT) and vision-language models (CLIP, BLIP). Despite substantial differences in training data, modality, and objectives, we observed robust cross-modal alignment. Action representations aligned strongly with decoder-only language models and BLIP (precision@15: 0.70-0.73), approaching the alignment observed among language models themselves. Alignment with CLIP and BERT was significantly weaker. These findings indicate that linguistic, visual, and action representations converge toward partially shared semantic structures, supporting modality-independent semantic organization and highlighting potential for cross-domain transfer in embodied AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑