VTok: 一种具有解耦空间-时间潜在变量的统一视频分词器
VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents
AI总结：
VTok通过解耦空间和时间潜在变量，实现紧凑且高效的视频分词，提升视频理解和生成任务的性能与效率。
摘要：
本工作提出了VTok，一种统一的视频分词框架，可用于生成和理解任务。不同于领先的视觉-语言系统通过简单的帧采样策略对视频进行分词，我们提出通过保留单个关键帧的空间特征，同时将每个后续帧编码为一个残差标记，从而解耦视频的空间和时间表示，实现紧凑且具有表达力的视频分词。我们的实验表明，VTok有效地将视频表示的复杂度从帧数与每帧标记数的乘积减少到它们的和，而残差标记充分捕捉了与关键帧相比的视角和运动变化。广泛的评估显示了VTok的有效性和效率：它在多种视频理解和文本到视频生成基准上均比使用简单分词的基线方法取得了显著更高的性能，所有情况下每个视频的标记序列都更短（例如，在我们的TV-Align基准上准确率提高了3.4%，在VBench得分上提高了1.9%）。值得注意的是，由于其更一致的时间编码，VTok在文本到视频生成中产生了更连贯的运动和更强的后续指导。我们希望VTok能成为未来视频理解和生成研究中标准化的视频分词范式。
英文摘要：
This work presents VTok, a unified video tokenization framework that can be used for both generation and understanding tasks. Unlike the leading vision-language systems that tokenize videos through a naive frame-sampling strategy, we propose to decouple the spatial and temporal representations of videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual token, achieving compact yet expressive video tokenization. Our experiments suggest that VTok effectively reduces the complexity of video representation from the product of frame count and per-frame token count to their sum, while the residual tokens sufficiently capture viewpoint and motion changes relative to the key frame. Extensive evaluations demonstrate the efficacy and efficiency of VTok: it achieves notably higher performance on a range of video understanding and text-to-video generation benchmarks compared with baselines using naive tokenization, all with shorter token sequences per video (e.g., 3.4% higher accuracy on our TV-Align benchmark and 1.9% higher VBench score). Remarkably, VTok produces more coherent motion and stronger guidance following in text-to-video generation, owing to its more consistent temporal encoding. We hope VTok can serve as a standardized video tokenization paradigm for future research in video understanding and generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
