SIFU：面向真实世界可用的穿衣人体重建的侧视条件隐式函数
SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction
AI总结：
SIFU提出侧视解耦Transformer与3D一致纹理细化流程，利用SMPL-X法线交叉注意力和扩散先验，从单张图像实现复杂姿态与宽松衣物的高精度穿衣人体重建，在几何与纹理上超越SOTA并支持3D打印等实际应用。
摘要：
从单张图像为真实世界应用创建高质量的穿衣人体3D模型至关重要。尽管近期取得了进展，但从自然场景图像中准确重建处于复杂姿态或穿着宽松衣物的人体，以及预测不可见区域的纹理，仍然是一个重大挑战。以往方法的一个关键局限在于，它们在从2D过渡到3D以及纹理预测方面缺乏足够的先验引导。为此，我们提出了SIFU（Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction，面向真实世界可用的穿衣人体重建的侧视条件隐式函数），这是一种结合了侧视解耦Transformer和3D一致纹理细化流程的新方法。SIFU在Transformer内部采用交叉注意力机制，以SMPL-X法线作为查询，在将2D特征映射到3D的过程中有效解耦侧视特征。该方法不仅提高了3D模型的精度，还增强了其鲁棒性，尤其是在SMPL-X估计不完美的情况下。我们的纹理细化流程利用基于文本到图像扩散的先验，为不可见视角生成逼真且一致的纹理。通过大量实验，SIFU在几何和纹理重建方面均超越了SOTA方法，在复杂场景中展现出更强的鲁棒性，并取得了前所未有的Chamfer和P2S测量结果。我们的方法可扩展到3D打印和场景构建等实际应用，展示了其在真实世界场景中的广泛实用性。项目主页：https://river-zhang.github.io/SIFU-projectpage/。
英文摘要：
Creating high-quality 3D models of clothed humans from single images for real-world applications is crucial. Despite recent advancements, accurately reconstructing humans in complex poses or with loose clothing from in-the-wild images, along with predicting textures for unseen areas, remains a significant challenge. A key limitation of previous methods is their insufficient prior guidance in transitioning from 2D to 3D and in texture prediction. In response, we introduce SIFU (Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction), a novel approach combining a Side-view Decoupling Transformer with a 3D Consistent Texture Refinement pipeline.SIFU employs a cross-attention mechanism within the transformer, using SMPL-X normals as queries to effectively decouple side-view features in the process of mapping 2D features to 3D. This method not only improves the precision of the 3D models but also their robustness, especially when SMPL-X estimates are not perfect. Our texture refinement process leverages text-to-image diffusion-based prior to generate realistic and consistent textures for invisible views. Through extensive experiments, SIFU surpasses SOTA methods in both geometry and texture reconstruction, showcasing enhanced robustness in complex scenarios and achieving an unprecedented Chamfer and P2S measurement. Our approach extends to practical applications such as 3D printing and scene building, demonstrating its broad utility in real-world scenarios. Project page https://river-zhang.github.io/SIFU-projectpage/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
