MRD：利用基于物理的可微渲染来探测视觉模型以理解3D场景
MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding
AI总结：
MRD通过基于物理的可微渲染技术，探测视觉模型对3D场景属性的隐式理解，揭示模型对形状、材质等物理参数的敏感性。
摘要：
尽管深度学习方法在许多视觉基准测试中取得了令人印象深刻的成功，但仍然难以理解和解释这些模型的表示和决策。虽然视觉模型通常在2D输入上进行训练，但通常假设它们发展出对底层3D场景的隐式表示（例如，对部分遮挡具有容忍性，或能够推断相对深度）。在这里，我们介绍了MRD（可微渲染的元体），一种利用基于物理的可微渲染来探测视觉模型对生成3D场景属性的隐式理解的方法，通过寻找在物理上不同但产生相同模型激活的3D场景参数（即模型元体）。与以前用于评估模型表示的像素基方法不同，这些重建结果始终基于物理场景描述。这意味着我们可以在保持材料和光照不变的情况下，探测模型对物体形状的敏感性。作为原理验证，我们评估了多个模型在恢复几何（形状）和双向反射分布函数（材料）场景参数方面的能力。结果表明，目标和优化场景之间的模型激活具有高度相似性，视觉结果各不相同。定性上，这些重建有助于研究模型对哪些物理场景属性敏感或不变。MRD有望通过使分析物理场景参数如何驱动模型响应的变化，推进我们对计算机视觉和人类视觉的理解。
英文摘要：
While deep learning methods have achieved impressive success in many vision benchmarks, it remains difficult to understand and explain the representations and decisions of these models. Though vision models are typically trained on 2D inputs, they are often assumed to develop an implicit representation of the underlying 3D scene (for example, showing tolerance to partial occlusion, or the ability to reason about relative depth). Here, we introduce MRD (metamers rendered differentiably), an approach that uses physically based differentiable rendering to probe vision models' implicit understanding of generative 3D scene properties, by finding 3D scene parameters that are physically different but produce the same model activation (i.e. are model metamers). Unlike previous pixel-based methods for evaluating model representations, these reconstruction results are always grounded in physical scene descriptions. This means we can, for example, probe a model's sensitivity to object shape while holding material and lighting constant. As a proof-of-principle, we assess multiple models in their ability to recover scene parameters of geometry (shape) and bidirectional reflectance distribution function (material). The results show high similarity in model activation between target and optimized scenes, with varying visual results. Qualitatively, these reconstructions help investigate the physical scene attributes to which models are sensitive or invariant. MRD holds promise for advancing our understanding of both computer and human vision by enabling analysis of how physical scene parameters drive changes in model responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
