It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/
Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);School of Computer Science, Wuhan University(武汉大学计算机学院);Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新 generation 人工智能技术及交叉应用重点实验室(东南大学),教育部,中国)
AlignGS: Aligning Geometry and Semantics for Robust Indoor Reconstruction from Sparse Views
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China(信息科学与电子工程学院,上海交通大学,上海,中国);Cooperative Mediant Innovation Center, Shanghai Jiao Tong University, Shanghai, China(协同医疗创新中心,上海交通大学,上海,中国);SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, Shanghai, China(上海交通大学巴黎精英技术学院)
LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning
City University of Hong Kong(香港城市大学);University of Science and Technology of China(中国科学技术大学);Utrecht University(乌特勒支大学);University of Electronic Science and Technology of China(电子科技大学)
Multimodal Industrial Anomaly Detection via Geometric Prior
Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院);Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)(教育部计算功率网络与信息安全重点实验室,山东计算机科学中心(济南国家超算中心),齐鲁工业大学(山东省科学院));Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(山东省计算功率互联网与服务计算重点实验室,山东省计算机科学基础研究中心)
EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors
University of California, Los Angeles, California, USA(加州大学洛杉矶分校);University of Cambridge, Cambridge, UK(剑桥大学);Technical University of Munich, Munich, Germany(慕尼黑技术大学)
GIGP: A Global Information Interacting and Geometric Priors Focusing Framework for Semi-supervised Medical Image Segmentation
The School of Mathematical Science, Capital Normal University(首都师范大学数学科学学院);The Academy for Multidisciplinary Studies, Capital Normal University(首都师范大学交叉科学研究院)
Leveraging Geometric Prior Uncertainty and Complementary Constraints for High-Fidelity Neural Indoor Surface Reconstruction
Department of Automation, Shanghai Jiao Tong University(自动化系,上海交通大学);Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong(机械与自动化工程系及T Stone机器人研究所,香港中文大学)
MGP-KAD: Multimodal Geometric Priors and Kolmogorov-Arnold Decoder for Single-View 3D Reconstruction in Complex Scenes
Doctoral Program in Empowerment Informatics, University of Tsukuba, Japan(赋能信息学博士项目,茨ukuba大学,日本);Center for Computational Science, Tsukuba, Ibaraki, Japan(计算科学中心,茨ukuba,Ibaraki,日本)
HumanRecon: Neural Reconstruction of Dynamic Human Using Geometric Cues and Physical Priors
Beijing University of Posts and Telecommunications(北京邮电大学);DJI Technology(大疆创新科技);Zhejiang University(浙江大学);Tampere Universities(坦佩雷大学);University of Adelaide(阿德莱德大学)