arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.19985cs.CVcs.AIcs.LG
收藏

嵌套专家混合:视觉 Token 的自适应处理

Mixture of Nested Experts: Adaptive Processing of Visual Tokens

Google DeepMind(谷歌DeepMind);University of Washington(华盛顿大学)

Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, Sujoy Paul

AI总结:

提出嵌套专家混合(MoNE)框架,利用嵌套专家结构按优先级动态选择视觉 token,在保持性能的同时将推理计算量减少两倍以上。

摘要:

视觉媒介(图像和视频)天然包含大量信息冗余,从而为在处理中利用效率提供了巨大机会。虽然基于 Vision Transformer (ViT) 的模型能有效地扩展到大规模数据场景,但它们未能利用这种固有的冗余,导致计算成本更高。Mixture of Experts (MoE) 网络在保持相同推理时间成本的同时展现出了可扩展性,但它们具有更大的参数占用空间。我们提出了 Mixture of Nested Experts (MoNE),它利用嵌套的专家结构,其中各个专家处于计算量与精度递增的曲线上。给定计算预算,MoNE 学习按优先级顺序动态选择 token,从而冗余的 token 通过成本更低的嵌套专家进行处理。使用此框架,我们实现了与基线模型相当的性能,同时将推理时间计算量减少了超过两倍。我们在标准图像和视频数据集 ImageNet-21K、Kinetics400 和 Something-Something-v2 上验证了我们的方法。我们进一步强调了 MoNE 的适应性,展示了它仅使用单一训练模型即可在视频的不同推理时间计算预算下保持强劲性能的能力。

英文摘要:

The visual medium (images and videos) naturally contains a large amount of information redundancy, thereby providing a great opportunity for leveraging efficiency in processing. While Vision Transformer (ViT) based models scale effectively to large data regimes, they fail to capitalize on this inherent redundancy, leading to higher computational costs. Mixture of Experts (MoE) networks demonstrate scalability while maintaining same inference-time costs, but they come with a larger parameter footprint. We present Mixture of Nested Experts (MoNE), which utilizes a nested structure for experts, wherein individual experts fall on an increasing compute-accuracy curve. Given a compute budget, MoNE learns to dynamically choose tokens in a priority order, and thus redundant tokens are processed through cheaper nested experts. Using this framework, we achieve equivalent performance as the baseline models, while reducing inference time compute by over two-fold. We validate our approach on standard image and video datasets - ImageNet-21K, Kinetics400, and Something-Something-v2. We further highlight MoNE$'$s adaptability by showcasing its ability to maintain strong performance across different inference-time compute budgets on videos, using only a single trained model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑