G3PT：通过跨尺度查询 Transformer 释放自回归建模在 3D 生成中的能力
G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer
AI总结：
本文提出 G3PT，通过多细节层次离散 token 与跨尺度查询 Transformer，使自回归建模适配无序 3D 数据，支持多条件 3D 形状生成并展现幂律缩放规律。
摘要：
自回归 Transformer 已经彻底改变了语言处理中的生成模型，并在图像和视频生成方面展现出巨大潜力。然而，当这些模型扩展到 3D 生成任务时，会面临重大挑战，因为它们依赖下一 token 预测来学习 token 序列，而这与 3D 数据的无序特性不兼容。本文没有对 3D 数据强加人为顺序，而是提出 G3PT，这是一种可扩展的由粗到细 3D 生成模型，采用跨尺度查询 Transformer。关键在于将基于点的 3D 数据映射为具有不同细节层次的离散 token，从而自然地在不同层次之间建立适合自回归建模的序列关系。此外，跨尺度查询 Transformer 在不同细节层次之间全局连接 token，而不需要有序序列。得益于这种方法，G3PT 具备一个通用的 3D 生成流程，可轻松支持多种条件结构，从而能够根据各种类型的条件生成 3D 形状。大量实验表明，与以往的 3D 生成方法相比，G3PT 实现了更优的生成质量和泛化能力。最重要的是，在 3D 生成领域，扩大 G3PT 首次揭示出明显的幂律缩放行为。
英文摘要：
Autoregressive transformers have revolutionized generative models in language processing and shown substantial promise in image and video generation. However, these models face significant challenges when extended to 3D generation tasks due to their reliance on next-token prediction to learn token sequences, which is incompatible with the unordered nature of 3D data. Instead of imposing an artificial order on 3D data, in this paper, we introduce G3PT, a scalable coarse-to-fine 3D generative model utilizing a cross-scale querying transformer. The key is to map point-based 3D data into discrete tokens with different levels of detail, naturally establishing a sequential relationship between different levels suitable for autoregressive modeling. Additionally, the cross-scale querying transformer connects tokens globally across different levels of detail without requiring an ordered sequence. Benefiting from this approach, G3PT features a versatile 3D generation pipeline that effortlessly supports diverse conditional structures, enabling the generation of 3D shapes from various types of conditions. Extensive experiments demonstrate that G3PT achieves superior generation quality and generalization ability compared to previous 3D generation methods. Most importantly, for the first time in 3D generation, scaling up G3PT reveals distinct power-law scaling behaviors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
