arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01437cs.CVcs.GR
收藏

Img2CAD:通过VLM辅助的条件分解从图像逆向工程三维CAD模型

Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization

Stanford University(斯坦福大学);NVIDIA(英伟达);Peking University(北京大学);DEVCOM Army Research Laboratory(陆军研究实验室)

Yang You, Mikaela Angelina Uy, Jiaqi Han, Rahul Thomas, Haotong Zhang, Yi Du, Hansheng Chen, Francis Engelmann, Suya You, Leonidas Guibas

AI总结:

本文提出Img2CAD方法,通过微调Llama3.2预测离散CAD结构,再用TrAssembler预测连续属性,实现从图像逆向生成三维CAD模型。

摘要:

从图像逆向工程三维计算机辅助设计(CAD)模型对于交互式编辑、制造、建筑、机器人等众多下游应用而言是一项重要任务。该任务的难点在于CAD输出与图像输入之间存在巨大的表示差异。CAD模型是精确的、程序化的构造,涉及将离散命令结构与连续属性相结合的序列化操作,这使得以端到端方式进行学习和优化极具挑战性。与此同时,输入图像引入了光度变化和传感器噪声等固有挑战,使逆向工程过程更加复杂。在这项工作中,我们提出了一种新颖的方法,将任务条件化地分解为两个子问题。首先,我们利用视觉语言基础模型(VLM),即一个经过微调的Llama3.2,来预测带有语义信息的全局离散基础结构。其次,我们提出了TrAssembler,它以带有语义的离散结构为条件,预测连续属性值。为了支持TrAssembler的训练,我们进一步构建了一个来自ShapeNet的常见物体标注CAD数据集。综合而言,我们的方法和数据展示了将自然图像CAD化的重要初步进展。代码和数据可在https://github.com/qq456cvb/Img2CAD获取。

英文摘要:

Reverse engineering 3D computer-aided design (CAD) models from images is an important task for many downstream applications including interactive editing, manufacturing, architecture, robotics, etc. The difficulty of the task lies in vast representational disparities between the CAD output and the image input. CAD models are precise, programmatic constructs that involve sequential operations combining discrete command structure with continuous attributes, making it challenging to learn and optimize in an end-to-end fashion. Concurrently, input images introduce inherent challenges such as photometric variability and sensor noise, complicating the reverse engineering process. In this work, we introduce a novel approach that conditionally factorizes the task into two sub-problems. First, we leverage vision-language foundation models (VLMs), a finetuned Llama3.2, to predict the global discrete base structure with semantic information. Second, we propose TrAssembler that, conditioned on the discrete structure with semantics, predicts the continuous attribute values. To support the training of our TrAssembler, we further constructed an annotated CAD dataset of common objects from ShapeNet. Putting all together, our approach and data demonstrate significant first steps towards CAD-ifying images in the wild. Code and data can be found in https://github.com/qq456cvb/Img2CAD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑