UniMoD: 基于深度混合的高效统一多模态Transformer
UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths
AI总结：
针对统一多模态Transformer训练成本高的问题，提出任务感知的token剪枝方法UniMoD，为每个任务设单独路由器，在Show-o和Emu3上分别减少15%和40%训练FLOPs，同时保持或提升性能。
摘要：
统一多模态Transformer在共享参数空间内处理生成和理解任务，近年来受到越来越多的关注。尽管已提出多种统一Transformer，但由于冗余token和繁重的注意力计算，训练这些模型成本高昂。过去，大语言模型研究表明，token剪枝方法（如深度混合MoD）可显著提升计算效率。MoD使用路由器在Transformer层内选择最重要的token进行处理。然而，直接将基于MoD的token剪枝应用于统一Transformer会导致性能次优，因为不同任务的token冗余程度不同。我们的工作通过（1）检查注意力权重模式，（2）评估层重要性和token冗余，（3）分析任务交互来分析统一Transformer。研究结果表明，token冗余主要受不同任务和层的影响。基于这些发现，我们引入UniMoD，一种任务感知的token剪枝方法，为每个任务采用单独的路由器来确定应剪枝哪些token。我们将该方法应用于Show-o和Emu3，在Show-o中减少约15%的训练FLOPs，在Emu3中减少约40%，同时在多个基准上保持或提高性能。代码将在https://github.com/showlab/UniMoD发布。
英文摘要：
Unified multimodal transformers, which handle both generation and understanding tasks within a shared parameter space, have received increasing attention in recent research. Although various unified transformers have been proposed, training these models is costly due to redundant tokens and heavy attention computation. In the past, studies on large language models have demonstrated that token pruning methods, such as Mixture of Depths (MoD), can significantly improve computational efficiency. MoD employs a router to select the most important ones for processing within a transformer layer. However, directly applying MoD-based token pruning to unified transformers will result in suboptimal performance because different tasks exhibit varying levels of token redundancy. In our work, we analyze the unified transformers by (1) examining attention weight patterns, (2) evaluating the layer importance and token redundancy, and (3) analyzing task interactions. Our findings reveal that token redundancy is primarily influenced by different tasks and layers. Building on these findings, we introduce UniMoD, a task-aware token pruning method that employs a separate router for each task to determine which tokens should be pruned. We apply our method to Show-o and Emu3, reducing training FLOPs by approximately 15% in Show-o and 40% in Emu3, while maintaining or improving performance on several benchmarks. Code will be released at https://github.com/showlab/UniMoD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
