arXivDaily arXiv每日学术速递 周一至周五更新
​通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2310.06836cs.CV

一种探测大型视觉模型对三维物理理解能力的通用协议

A General Protocol to Probe Large Vision Models for 3D Physical Understanding

University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；Coop. Medianet Innovation Center（Coop. Medianet创新中心）

Guanqi Zhan, Chuanxia Zheng, Weidi Xie, Andrew Zisserman

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种通用轻量协议，通过在大型视觉模型特征上训练判别分类器来探测其对三维场景物理属性的理解能力，并发现Stable Diffusion和DINOv2在多数属性上表现最佳。

摘要：

本文的目标是探测大型视觉模型，以确定它们在多大程度上“理解”图像中所描绘三维场景的不同物理属性。为此，我们做出了以下贡献：（i）我们引入了一种通用且轻量化的协议，通过在现成大型视觉模型的特征上训练针对这些属性的判别分类器，来评估这些特征是否编码了三维场景的若干物理“属性”。这些探针被应用于带有属性标注的真实图像数据集上。（ii）我们将该协议应用于涵盖场景几何、场景材质、支撑关系、光照以及视角相关度量的属性，并应用于包括CLIP、DINOv1、DINOv2、VQGAN、Stable Diffusion在内的大型视觉模型。（iii）我们发现，Stable Diffusion和DINOv2的特征在若干属性的判别学习上表现良好，包括场景几何、支撑关系、阴影和深度，但在遮挡和材质方面表现较差，同时在所有属性上均优于DINOv1、CLIP和VQGAN。（iv）我们观察到，Stable Diffusion特征的不同时间步，以及DINO/CLIP/VQGAN的不同Transformer层，在不同属性上各有所长，这为三维物理理解的潜在应用开辟了可能。

英文摘要：

Our objective in this paper is to probe large vision models to determine to what extent they 'understand' different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical 'properties' of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑