LongSSM:论语言建模中状态空间模型的长度扩展
LongSSM: On the Length Extension of State-space Models in Language Modelling
AI总结:
本文研究状态空间模型的长度扩展问题,发现零隐藏状态初始化导致长度扩展困难,指出其等价于多项式外推,并提出改变隐藏状态初始化方案的方法,有效提升了长度扩展能力并实现长记忆模型的高效训练。
摘要:
本文研究语言建模中状态空间模型(SSMs)的长度扩展。长度扩展是指在短序列上训练模型,并在更长序列上测试。我们证明使用零隐藏状态初始化训练的状态空间模型在长度扩展上存在困难。我们指出长度扩展等价于多项式外推,以此解释了该困难。基于该理论,我们提出一种简单有效的方法——改变隐藏状态初始化方案——来改善长度扩展。此外,我们的方法表明,使用长训练序列长度对长度扩展有益但并非必要。改变隐藏状态初始化能用较小的训练上下文长度实现长记忆模型的高效训练。
英文摘要:
In this paper, we investigate the length-extension of state-space models (SSMs) in language modeling. Length extension involves training models on short sequences and testing them on longer ones. We show that state-space models trained with zero hidden states initialization have difficulty doing length extension. We explain this difficulty by pointing out the length extension is equivalent to polynomial extrapolation. Based on the theory, we propose a simple yet effective method - changing the hidden states initialization scheme - to improve the length extension. Moreover, our method shows that using long training sequence length is beneficial but not necessary to length extension. Changing the hidden state initialization enables the efficient training of long-memory model with a smaller training context length.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。