发表机构
Pusan National University; Shanghai Jiao Tong University; Hanyang University（釜山国立大学; 上海交通大学; 汉阳大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对音频语言模型多技能联合训练中的干扰问题，提出SkillFormer，通过技能特定低秩适配器与学习路由器实现技能分解与动态组合，以不到4%的额外参数在三个基准上将平均准确率提升2.5至4.1个百分点。
AI 中文摘要
音频语言模型必须处理数十种不同的技能，从音高比较和说话人计数到音乐节奏估计和情感识别。在所有技能上同时进行联合训练会导致干扰：一项技能上的提升往往以另一项技能的损失为代价。我们提出了SkillFormer，它将音频理解分解为技能特定的低秩适配器，并在推理时通过一个学习到的路由器进行组合。该路由器检查问题以决定激活哪些适配器以及每个适配器应承担多少权重，从而使音高查询与体裁分类查询使用不同的参数。一种交替训练计划在联合校准路由器之前，先在各自的技能簇上更新每个适配器，从而防止标准多任务优化中出现的梯度冲突。SkillFormer增加的参数不到基础模型参数的4%，并且不需要对音频编码器或语言主干进行任何更改。在MMSU、MMAU-Pro和MMAR上对三个架构不同的模型进行评估，它将平均准确率提高了2.5到4.1个百分点，并且在感知、推理和语义子类别上取得了均衡的提升。
英文摘要
Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbf{SkillFormer}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to activate and how much weight each should carry, so that a pitch query engages different parameters than a genre classification query. An alternating training schedule updates each adapter on its own skill cluster before jointly calibrating the router, preventing the gradient conflicts that arise in standard multi-task optimization. SkillFormer adds fewer than 4\% of the base model's parameters and requires no changes to the audio encoder or language backbone. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, it raises the average accuracy by 2.5 to 4.1 points, with balanced gains across perception, reasoning, and semantic subcategories.
