arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.03120cs.CV
收藏

CAT:内容自适应图像分词

CAT: Content-Adaptive Image Tokenization

Carnegie Mellon University(卡内基梅隆大学);Meta

Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, Chunting Zhou

AI总结:

本文提出内容自适应图像分词器 CAT,利用 LLM 依据图像描述预测复杂度并动态分配 token,在重建和 ImageNet DiT 生成中提升 FID 与 18.5% 推理吞吐量。

摘要:

大多数现有图像分词器将图像编码为固定数量的 token 或图像块,忽视了图像复杂度的内在差异。为解决这一问题,我们提出内容自适应分词器(Content-Adaptive Tokenizer,CAT),它根据图像内容动态调整表示容量,并将较简单的图像编码为更少的 token。我们设计了一种基于描述的评估系统,利用大型语言模型(LLM)预测内容复杂度,并在考虑对人类感知至关重要的因素后,为给定图像确定最佳压缩比。CAT 在具有多种压缩比的图像上进行训练,在图像重建中表现出稳健性能。我们还利用其变长潜在表示训练 Diffusion Transformers(DiT)以进行 ImageNet 生成。通过优化 token 分配,相比在相同浮点运算量下训练的固定压缩比基线,CAT 改善了 FID 分数,并将推理吞吐量提高了 18.5%。

英文摘要:

Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑