arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.06356cs.LGcs.CLcs.CV
收藏

Monkey Jump : 基于MoE的参数高效微调方法用于高效多任务学习

Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning

UCF(UCF大学);Nokia Bell Labs(诺基亚贝尔实验室)

Nusrat Jahan Prottasha, Md Kowsher, Chun-Nam Yu, Chen Chen, Ozlem Garibay

AI总结:

Monkey Jump通过隐式专家路由实现高效多任务学习,无需额外参数,提升表达性并降低训练成本。

摘要:

混合专家变种的参数高效微调方法能够实现逐token专业化,但它们引入了额外的可训练路由和专家参数,增加了内存使用和训练成本。这 undermines 参数高效微调的核心目标。我们提出 Monkey Jump,一种将混合专家式专业化引入参数高效微调的方法,无需引入额外的可训练参数用于专家或路由。代替添加新的适配器作为专家,Monkey Jump将每个Transformer块中已有的适配器(如查询、键、值、上和下投影)视为隐式专家,并在其中路由token。路由使用k-means聚类与指数移动平均的聚类中心进行,无需梯度和无需学习参数。我们理论证明,逐token路由增加表达性,并能通过避免抵消效应优于共享适配器。在涵盖14个文本、14个图像和19个视频基准的多任务实验中,Monkey Jump在使用7到29倍更少的可训练参数、高达48%更低的内存消耗以及1.5到2倍更快的训练速度的情况下,实现了与基于混合专家的参数高效微调方法相竞争的性能。Monkey Jump是架构无关的,可以应用于任何基于适配器的参数高效微调方法。

英文摘要:

Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This undermines the core goal of parameter-efficient fine-tuning. We propose Monkey Jump, a method that brings mixture-of-experts-style specialization to parameter-efficient fine-tuning without introducing extra trainable parameters for experts or routers. Instead of adding new adapters as experts, Monkey Jump treats the adapters already present in each Transformer block (such as query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed using k-means clustering with exponentially moving averaged cluster centers, requiring no gradients and no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. Across multi-task experiments covering 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with mixture-of-experts-based parameter-efficient fine-tuning methods while using 7 to 29 times fewer trainable parameters, up to 48 percent lower memory consumption, and 1.5 to 2 times faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based parameter-efficient fine-tuning method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑