arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.18830cs.CV
收藏

空间-时间状态传播自回归模型用于4D物体生成

Spatial-Temporal State Propagation Autoregressive Model for 4D Object Generation

Macau University of Science and Technology(澳门科技大学);TeleAI;Peking University(北京大学)

Liying Yang, Jialun Liu, Jiakui Hu, Chenhao Guan, Haibin Huang, Fangqiu Yi, Chi Zhang, Yanyan Liang

AI总结:

4DSTAR通过空间-时间状态传播自回归模型和4D VQ-VAE生成具有时间-空间一致性的4D物体,实现高质量的4D物体生成。

摘要:

生成高质量的4D物体并保持空间-时间一致性仍然具有挑战性。现有的扩散方法往往在空间-时间一致性上遇到困难,因为它们未能利用所有先前时间步的输出来指导当前时间步的生成。因此,我们提出了一个空间-时间状态传播自回归模型(4DSTAR),该模型生成保持时间-空间一致性的4D物体。4DSTAR将生成问题建模为预测代表4D物体的token。它包含两个关键组成部分:(1)动态空间-时间状态传播自回归模型(STAR)被提出,实现了空间-时间一致的生成。不同于标准自回归模型,STAR根据时间步将预测token分成组。它通过传播来自先前组的空间-时间状态来建模长期依赖性,并利用这些依赖性来指导下一步的生成。为此,提出了一种空间-时间容器,它动态更新来自所有历史组的有效空间-时间状态特征,然后更新的特征作为条件特征来指导下一个token组的预测。(2)提出了4D VQ-VAE,它隐式地将4D结构编码到离散空间,并将STAR预测的离散token解码为时间一致的动态3D高斯分布。实验表明,4DSTAR生成具有空间-时间一致性的4D物体,并且在性能上与扩散模型相媲美。

英文摘要:

Generating high-quality 4D objects with spatial-temporal consistency is still formidable. Existing diffusion-based methods often struggle with spatial-temporal inconsistency, as they fail to leverage outputs from all previous timesteps to guide the generation at the current timestep. Therefore, we propose a Spatial-Temporal State Propagation AutoRegressive Model (4DSTAR), which generates 4D objects maintaining temporal-spatial consistency. 4DSTAR formulates the generation problem as the prediction of tokens that represent the 4D object. It consists of two key components: (1) The dynamic spatial-temporal state propagation autoregressive model (STAR) is proposed, which achieves spatial-temporal consistent generation. Unlike standard autoregressive models, STAR divides prediction tokens into groups based on timesteps. It models long-term dependencies by propagating spatial-temporal states from previous groups and utilizes these dependencies to guide generation at the next timestep. To this end, a spatial-temporal container is proposed, which dynamically updating the effective spatial-temporal state features from all historical groups, then updated features serve as conditional features to guide the prediction of the next token group. (2) The 4D VQ-VAE is proposed, which implicitly encodes the 4D structure into discrete space and decodes the discrete tokens predicted by STAR into temporally coherent dynamic 3D Gaussians. Experiments demonstrate that 4DSTAR generates spatial-temporal consistent 4D objects, and achieves performance competitive with diffusion models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑