arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.02473cs.CV
收藏

基于差分运动的形状与材质生成式感知

Generative Perception of Shape and Material from Differential Motion

Harvard University(哈佛大学);Kyoto University(京都大学)

Xinran Nicole Han, Ko Nishino, Todd Zickler

AI总结:

受人类借助微动消解视觉歧义的行为启发,本文提出条件去噪扩散模型,从差分运动短视频中生成解耦的形状与材质映射,可捕捉静态观测歧义、在运动时输出更准确估计,为具身系统视觉推理优化提供思路。

摘要:

从单张图像感知物体的形状与材质本身存在固有歧义,在光照未知且不受约束的场景下尤为明显。尽管存在该问题,人类通常能够区分形状与材质,当判断不确定时,往往会轻微移动头部或转动物体来消解歧义。受该行为启发,本文提出一种新型条件去噪扩散模型,可从物体做差分运动的短视频中生成形状-材质映射样本。该参数高效架构支持直接在像素空间训练,可同时生成物体的多个解耦属性。模型在数量适中的合成物体运动视频上训练,以形状和材质为监督信号,展现出极具说服力的涌现行为:对于静态观测,模型可生成多样、多模态的合理形状-材质映射预测,捕捉固有歧义;当物体运动时,预测分布会收敛至更准确的解释。该模型还能对歧义性更低的真实世界物体生成高质量的形状与材质估计结果。通过从单视图观测拓展至连续运动观测,并利用生成式感知捕捉视觉歧义,本研究为提升具身物理系统的视觉推理能力提供了可行方向。

英文摘要:

Perceiving the shape and material of an object from a single image is inherently ambiguous, especially when lighting is unknown and unconstrained. Despite this, humans can often disentangle shape and material, and when they are uncertain, they often move their head slightly or rotate the object to help resolve the ambiguities. Inspired by this behavior, we introduce a novel conditional denoising-diffusion model that generates samples of shape-and-material maps from a short video of an object undergoing differential motions. Our parameter-efficient architecture allows training directly in pixel-space, and it generates many disentangled attributes of an object simultaneously. Trained on a modest number of synthetic object-motion videos with supervision on shape and material, the model exhibits compelling emergent behavior: For static observations, it produces diverse, multimodal predictions of plausible shape-and-material maps that capture the inherent ambiguities; and when objects move, the distributions converge to more accurate explanations. The model also produces high-quality shape-and-material estimates for less ambiguous, real-world objects. By moving beyond single-view to continuous motion observations, and by using generative perception to capture visual ambiguities, our work suggests ways to improve visual reasoning in physically-embodied systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑