arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.10282cs.CVcs.AIcs.CLcs.LG
收藏

狗仔队:深入研究语言和视觉模型的基础观点描述能力

Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions

University of Jena(耶拿大学);University of Magdeburg(马格德堡大学);University of Bielefeld(比勒费尔德大学)

Henrik Voigt, Jan Hombeck, Monique Meuschke, Kai Lawonn, Sina Zarrieß

摘要:

现有的语言和视觉模型在图像-文本理解方面取得了令人印象深刻的性能。然而,这是一个开放的问题,在何种程度上,他们可以用于语言理解在3D环境中,以及他们是否隐含地获取3D对象的知识,例如,关于不同的意见,一个对象。在本文中,我们调查是否一个国家的最先进的语言和视觉模型,CLIP,是能够地面的3D对象的透视图描述,并确定基于文本查询的常见对象的规范视图。我们提出了一个评估框架,使用一个环绕的相机周围的3D对象从不同的角度生成图像,并评估他们的相似性自然语言描述。我们发现,预训练的CLIP模型在大多数规范视图上表现不佳,即使在可用训练数据很少的条件下,使用硬负采样和随机对比进行微调也会产生良好的结果。

英文摘要:

Existing language and vision models achieve impressive performance in image-text understanding. Yet, it is an open question to what extent they can be used for language understanding in 3D environments and whether they implicitly acquire 3D object knowledge, e.g. about different views of an object. In this paper, we investigate whether a state-of-the-art language and vision model, CLIP, is able to ground perspective descriptions of a 3D object and identify canonical views of common objects based on text queries. We present an evaluation framework that uses a circling camera around a 3D object to generate images from different viewpoints and evaluate them in terms of their similarity to natural language descriptions. We find that a pre-trained CLIP model performs poorly on most canonical views and that fine-tuning using hard negative sampling and random contrasting yields good results even under conditions with little available training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑