arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.09592cs.CVcs.AI
收藏

ControLRM:基于大型重建模型的快速可控3D生成

ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Model

South China University of Technology(华南理工大学);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)

Hongbin Xu, Weitao Chen, Zhipeng Zhou, Feng Xiao, Baigui Sun, Mike Zheng Shou, Wenxiong Kang

AI总结:

针对3D生成可控性差且耗时长的问题,提出端到端前馈模型ControLRM,通过联合训练框架复用预训练LRM编码层并解锁三平面解码器,实现2D与3D表示的隐式对齐,在多数据集上展现出强大的快速可控生成与泛化能力。

摘要:

尽管3D生成方法近期取得了进展,但实现可控性仍是一个难题。当前使用分数蒸馏采样的方法受制于耗时的繁琐流程。此外,先生成2D表示再映射到3D的过程缺乏两种表示形式间的内部对齐。为应对这些挑战,我们提出了ControLRM,这是一种端到端前馈模型,旨在利用大型重建模型(LRM)实现快速可控的3D生成。ControLRM由2D条件生成器、条件编码Transformer和三平面解码器Transformer组成。我们不从零开始训练模型,而是采用联合训练框架。在条件训练分支中,我们锁定三平面解码器,并复用LRM中用数百万3D数据预训练的深层且稳健的编码层。在图像训练分支中,我们解锁三平面解码器,以建立2D和3D表示之间的隐式对齐。为确保评估的客观性,我们从三个不同数据集(G-OBJ、GSO、ABO)中精选评估样本,而非依赖人工挑选的生成结果。对3D可控性和生成质量的定量与定性比较实验表明,我们提出的方法具有强大的泛化能力。

英文摘要:

Despite recent advancements in 3D generation methods, achieving controllability still remains a challenging issue. Current approaches utilizing score-distillation sampling are hindered by laborious procedures that consume a significant amount of time. Furthermore, the process of first generating 2D representations and then mapping them to 3D lacks internal alignment between the two forms of representation. To address these challenges, we introduce ControLRM, an end-to-end feed-forward model designed for rapid and controllable 3D generation using a large reconstruction model (LRM). ControLRM comprises a 2D condition generator, a condition encoding transformer, and a triplane decoder transformer. Instead of training our model from scratch, we advocate for a joint training framework. In the condition training branch, we lock the triplane decoder and reuses the deep and robust encoding layers pretrained with millions of 3D data in LRM. In the image training branch, we unlock the triplane decoder to establish an implicit alignment between the 2D and 3D representations. To ensure unbiased evaluation, we curate evaluation samples from three distinct datasets (G-OBJ, GSO, ABO) rather than relying on cherry-picking manual generation. The comprehensive experiments conducted on quantitative and qualitative comparisons of 3D controllability and generation quality demonstrate the strong generalization capacity of our proposed approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑