arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.14760cs.CV
收藏

3D视觉-语言模型真的能理解自然语言吗?

Can 3D Vision-Language Models Truly Understand Natural Language?

The University of Hong Kong(香港大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Weipeng Deng, Jihan Yang, Runyu Ding, Jiahui Liu, Yijiang Li, Xiaojuan Qi, Edith Ngai

AI总结:

本文针对3D视觉-语言模型对语言风格变体敏感的问题,提出语言鲁棒性评估任务和3D语言鲁棒性数据集,系统揭示了现有模型性能显著下降及融合模块脆弱有偏的根源,并设计了一个由LLM驱动的免训练模块以提升语言鲁棒性。

摘要:

3D视觉-语言(3D-VL)任务的快速发展为人类通过自然语言与具身智能体或机器人交互开辟了新途径。尽管取得了这些进展,我们发现了一个显著局限:现有3D-VL模型对语言输入的风格表现出敏感性,难以理解语义相同但以不同变体书写的句子。这一观察引出了一个关键问题:3D视觉-语言模型真的能理解自然语言吗?为了测试3D-VL模型的语言理解能力,我们首先提出了一个语言鲁棒性任务,用于在各种任务中系统评估3D-VL模型,在呈现不同语言风格变体时对其性能进行基准测试。重要的是,由于人类语言的多样性和不可预测性,这些变体常见于需要与人类直接交互的应用中,例如具身机器人。我们提出了一个基于人类语言特征设计的3D语言鲁棒性数据集,以促进对鲁棒性的系统研究。我们的综合评估揭示了所有现有模型在各种3D-VL任务中的性能显著下降。即使是最先进的3D-LLM也无法理解同一句子的一些变体。进一步的深入分析表明,现有模型具有脆弱且有偏的融合模块,这源于现有数据集的低多样性。最后,我们提出了一个由LLM驱动的免训练模块,以提高语言鲁棒性。数据集和代码将在github上提供。

英文摘要:

Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new avenues for human interaction with embodied agents or robots using natural language. Despite this progress, we find a notable limitation: existing 3D-VL models exhibit sensitivity to the styles of language input, struggling to understand sentences with the same semantic meaning but written in different variants. This observation raises a critical question: Can 3D vision-language models truly understand natural language? To test the language understandability of 3D-VL models, we first propose a language robustness task for systematically assessing 3D-VL models across various tasks, benchmarking their performance when presented with different language style variants. Importantly, these variants are commonly encountered in applications requiring direct interaction with humans, such as embodied robotics, given the diversity and unpredictability of human language. We propose a 3D Language Robustness Dataset, designed based on the characteristics of human language, to facilitate the systematic study of robustness. Our comprehensive evaluation uncovers a significant drop in the performance of all existing models across various 3D-VL tasks. Even the state-of-the-art 3D-LLM fails to understand some variants of the same sentences. Further in-depth analysis suggests that the existing models have a fragile and biased fusion module, which stems from the low diversity of the existing dataset. Finally, we propose a training-free module driven by LLM, which improves language robustness. Datasets and code will be available at github.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑