DeRA: 解耦表示对齐用于视频标记化
DeRA: Decoupled Representation Alignment for Video Tokenization
AI总结：
DeRA通过解耦时空表示学习，提升视频标记化效率和性能，并在视频生成任务中取得新突破。
摘要：
本文提出了DeRA，一种新型的1D视频标记器，通过解耦视频标记化中的时空表示学习，以实现更好的训练效率和性能。具体而言，DeRA维护一个紧凑的1D潜在空间，将视频编码分解为外观和运动流，这些流与预训练的视觉基础模型对齐，以分别捕捉视频中的空间语义和时间动态。为了解决异构监督引入的梯度冲突，我们进一步提出了对称对齐-冲突投影（SACP）模块，该模块通过抑制冲突方向上的组件来主动重新公式化梯度。广泛的实验表明，DeRA在UCF-101数据集上以rFVD指标优于先前最先进的视频标记器LARP 25%。此外，使用DeRA进行自回归视频生成，在UCF-101类别条件生成和K600帧预测上也取得了新的最先进结果。
英文摘要：
This paper presents DeRA, a novel 1D video tokenizer that decouples the spatial-temporal representation learning in video tokenization to achieve better training efficiency and performance. Specifically, DeRA maintains a compact 1D latent space while factorizing video encoding into appearance and motion streams, which are aligned with pretrained vision foundation models to capture the spatial semantics and temporal dynamics in videos separately. To address the gradient conflicts introduced by the heterogeneous supervision, we further propose the Symmetric Alignment-Conflict Projection (SACP) module that proactively reformulates gradients by suppressing the components along conflicting directions. Extensive experiments demonstrate that DeRA outperforms LARP, the previous state-of-the-art video tokenizer by 25% on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressive video generation, we also achieve new state-of-the-art results on both UCF-101 class-conditional generation and K600 frame prediction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
