arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.26290cs.LGcs.AI
收藏

通过Hankel降阶建模的SSM适配器:注入位置决定长上下文微调中的任务适用性

SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

Hitachi America Ltd.(日立美洲有限公司)

Omanshu Thapliyal

AI总结:

提出Hankel降阶模型(HRM)适配器,一种基于SSM的残差模块,通过平衡截断初始化,在长上下文任务中优于LoRA,并支持FFT并行扫描。

摘要:

虽然参数高效微调(PEFT)通常针对注意力投影器,但其在需要序列状态累积的任务中的有效性尚未充分探索。我们研究了PEFT在此类任务中是否能从状态空间模型(SSM)适配器中受益,以及MLP块是否是更好的注入位置。我们引入了Hankel降阶模型(HRM)适配器,这是一种基于SSM的残差模块,通过经验Hankel Gramians的平衡截断初始化。通过利用系统矩阵$\ar{A}$的时间不变性,HRM实现了基于FFT的精确并行扫描,在所有上下文长度上达到与LoRA相同的计算效率。在Mistral-7B(840万可训练参数)的等参数评估中,HRM在LongBench任务上优于LoRA变体,包括QuALITY(相对准确率提升34.8%)和QMSum(相对ROUGE-1提升71.6%)。HRM在18种合成状态跟踪(DFA、Parity)和字符级语言建模(enwik8)配置中进一步展现出持续优势。门控分析表明,HRM适配器有效地学习调节循环,为长上下文序列建模提供了低秩适应的稳健架构替代方案。

英文摘要:

While parameter-efficient fine-tuning (PEFT) typically targets attention projectors, its efficacy for tasks requiring sequential state accumulation remains under-explored. We examine if PEFT for such tasks can benefit from state space model (SSMs) adapters, and if MLP blocks are better injection sites. We introduce Hankel Reduced order Model (HRM) adapter, an SSM-based residual module initialized via Balanced Truncation of empirical Hankel Grammians. By leveraging the time-invariance of the system matrix $\bar{A}$, HRM enables an exact FFT-based parallel scan, achieving computational parity with LoRA across all context lengths. In iso-parametric evaluations on Mistral-7B (8.4M trainable parameters), HRM outperforms LoRA variants on LongBench tasks, including QuALITY (+34.8\% relative accuracy) and QMSum (+71.6\% relative ROUGE-1). HRM further demonstrates consistent superiority across 18 configurations of synthetic state-tracking (DFA, Parity) and character-level language modeling (enwik8). Gate analysis reveals that HRM adapters effectively learn to modulate recurrence, providing a robust architectural alternative to low-rank adaptation for long-context sequence modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑