arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04449cs.CVcs.CL
收藏

p-MoD：通过渐进比例衰减构建混合深度多模态大语言模型

p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay

State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；China Mobile Research Institute（中国移动研究院）；Shanghai AI Lab（上海AI实验室）

Jun Zhang, Desen Meng, Zhengming Zhang, Zhenpeng Huang, Tao Wu, Limin Wang

备注：
Accepted by ICCV 2025; Code released at https://github.com/MCG-NJU/p-MoD
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出p-MoD架构，通过混合深度机制和渐进比例衰减策略，在保持性能的同时显著降低多模态大语言模型的训练和推理成本。

摘要：

尽管多模态大语言模型（MLLMs）在各类任务中表现出色，但其高昂的训练和推理成本阻碍了其进一步发展。本文提出p-MoD，一种高效的多模态大语言模型架构，在保持模型性能的同时显著降低训练和推理成本。多模态大语言模型中的大部分计算源于基于Transformer的大语言模型所处理的海量视觉令牌。为此，我们利用混合深度（MoD）机制，每个大语言模型层选择必要的视觉令牌进行处理，同时跳过冗余令牌。然而，将MoD集成到多模态大语言模型中并非易事。为应对训练和推理稳定性以及训练数据有限等挑战，我们通过两种新颖设计对MoD模块进行适配：tanh门控权重归一化（TanhNorm）和对称令牌重新加权（STRing）。此外，我们观察到视觉令牌在更深层表现出更高的冗余性，因此设计了渐进比例衰减（PRD）策略，该策略采用移位余弦调度，逐层逐渐降低令牌保留比例。这一关键设计充分释放了MoD的潜力，显著提升了我们模型的效率和性能。在两个基线模型上的大量实验（覆盖15个基准）表明，我们的模型性能与相应基线相当甚至更优，同时推理时仅需55.6%的TFLOPs和53.7%的KV缓存存储，训练时仅需77.7%的GPU小时。

英文摘要：

Despite the remarkable performance of multimodal large language models (MLLMs) across diverse tasks, the substantial training and inference costs impede their advancement. In this paper, we propose p-MoD, an efficient MLLM architecture that significantly reduces training and inference costs while maintaining model performance. The majority of computation in MLLMs stems from the overwhelming volume of vision tokens processed by the transformer-based LLM. Accordingly, we leverage the Mixture-of-Depths (MoD) mechanism, where each LLM layer selects essential vision tokens to process while skipping redundant ones. However, integrating MoD into MLLMs is non-trivial. To address the challenges of training and inference stability as well as limited training data, we adapt the MoD module with two novel designs: tanh-gated weight normalization (TanhNorm) and symmetric token reweighting (STRing). Moreover, we observe that vision tokens exhibit higher redundancy in deeper layers and thus design a progressive ratio decay (PRD) strategy, which gradually reduces the token retention ratio layer by layer, employing a shifted cosine schedule. This crucial design fully unleashes the potential of MoD, significantly boosting the efficiency and performance of our models. Extensive experiments on two baseline models across 15 benchmarks show that our model matches or even surpasses the performance of corresponding baselines, while requiring only 55.6% TFLOPs and 53.7% KV cache storage during inference, and 77.7% GPU hours during training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑