ElasticTok:面向图像和视频的自适应词元化
ElasticTok: Adaptive Tokenization for Image and Video
AI总结:
提出ElasticTok方法,通过基于先前帧的自适应掩码技术将图像和视频动态编码为可变数量词元,解决了固定词元编码的效率瓶颈,为高效的多模态模型和智能体发展奠定基础。
摘要:
高效的视频词元化仍然是学习能够处理长视频序列的通用视觉模型的关键瓶颈。现有方法受限于将视频编码为固定数量的词元,词元过少会导致编码信息过度丢失,词元过多则会导致序列长度过长而难以处理。在本研究中,我们提出了ElasticTok,一种基于先前帧自适应地将单帧编码为可变数量词元的方法。为了以计算上可扩展的方式实现这一点,我们提出了一种掩码技术,在每帧词元编码的末尾丢弃随机数量的词元。在推理阶段,ElasticTok能够按需动态分配词元——更复杂的数据可以使用更多词元,而较简单的数据只需少量词元。我们在图像和视频上的实证评估证明了本方法在高效词元使用方面的有效性,为未来开发更强大的多模态模型、世界模型和智能体铺平了道路。
英文摘要:
Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。