HarmonyView:单图转3D中的一致性与多样性协调
HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D
AI总结:
HarmonyView提出一种扩散采样技术,在单图转3D中同时优化多视角一致性与新视角多样性,并引入CLIP-based评估指标,实现两者双赢。
摘要:
近期在单图像3D生成方面的进展凸显了多视角一致性的重要性,其利用在互联网规模图像上预训练的大规模扩散模型中的3D先验知识。然而,由于将2D图像转换为3D内容存在歧义,可能产生众多潜在形状,新视角多样性这一方面在研究领域中仍未得到充分探索。在此,我们旨在通过同时处理一致性和多样性来填补这一研究空白。然而,由于这两个方面固有的权衡,在它们之间取得平衡构成相当大的挑战。这项工作引入了HarmonyView,一种简单而有效的扩散采样技术,擅长分解单图像3D生成中的两个复杂方面:一致性和多样性。该方法为在采样过程中更细致地探索这两个关键维度铺平了道路。此外,我们提出了一种基于CLIP图像和文本编码器的新评估指标,以全面评估生成视角的多样性,该指标与人类评估者的判断高度一致。在实验中,HarmonyView实现了和谐的平衡,在一致性和多样性方面均展现出双赢的局面。
英文摘要:
Recent progress in single-image 3D generation highlights the importance of multi-view coherency, leveraging 3D priors from large-scale diffusion models pretrained on Internet-scale images. However, the aspect of novel-view diversity remains underexplored within the research landscape due to the ambiguity in converting a 2D image into 3D content, where numerous potential shapes can emerge. Here, we aim to address this research gap by simultaneously addressing both consistency and diversity. Yet, striking a balance between these two aspects poses a considerable challenge due to their inherent trade-offs. This work introduces HarmonyView, a simple yet effective diffusion sampling technique adept at decomposing two intricate aspects in single-image 3D generation: consistency and diversity. This approach paves the way for a more nuanced exploration of the two critical dimensions within the sampling process. Moreover, we propose a new evaluation metric based on CLIP image and text encoders to comprehensively assess the diversity of the generated views, which closely aligns with human evaluators' judgments. In experiments, HarmonyView achieves a harmonious balance, demonstrating a win-win scenario in both consistency and diversity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。