arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.30154cs.CLcs.LG
收藏

baseVultra:具备容量重分配的紧凑非线性循环单元

baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

Southwest University(西南大学)

Linke Zhang

AI总结:

提出baseVultra紧凑循环单元,将循环预算集中于候选生成矩阵,在紧凑设置下的序列建模任务中性能优于参数匹配的LSTM、GRU,大尺寸场景下与全门单元相当。

摘要:

非线性循环网络在流式序列建模中仍具实用性,但门控单元设计仍被LSTM和GRU主导。本研究探讨是否可将循环计算从门变换重分配至候选生成。我们提出baseVultra,一种紧凑循环单元,其将密集循环预算集中于单个H×H候选生成矩阵,同时用O(H)对角交互替代循环门矩阵。候选更新结合该密集分支、两个低秩辅助分支、EMA缓存及可学习输出融合;门控还通过对角项获取对h_{t-2}的轻量二阶访问。在TinyShakespeare字符级语言建模中,baseVultra在紧凑单层200K参数设置下验证损失为1.594,而参数匹配的LSTM为1.604、GRU为1.616,且该配置下匹配隐藏尺寸时其密集循环矩阵向量计算量约为GRU的54%。在约大45倍的基准text8上,baseVultra与全门单元性能相当(每字符比特数1.854,对应GRU为1.849、LSTM为1.837)。在形式语言探测任务中,最大增益出现在隐藏状态容量受限的场景(H=16时Counter任务的OOD2准确率为0.991,对应GRU为0.761、LSTM为0.691),且随隐藏尺寸增大该增益消失。这些结果明确了以候选为中心的循环容量分配的适用场景与时机:效益集中于紧凑 regime,而全门单元在规模较大时仍具竞争力。

英文摘要:

Nonlinear recurrent networks remain useful for streaming sequence modeling, yet gated cell design is still dominated by LSTM and GRU. We study whether recurrent computation can be reallocated away from gate transforms and toward candidate generation. We propose baseVultra, a compact recurrent cell that concentrates the dense recurrent budget on a single H x H candidate-generation matrix, while replacing recurrent gate matrices with O(H) diagonal interactions. The candidate update combines this dense branch with two low-rank auxiliary branches, an EMA cache, and learnable output blending; the gates additionally receive lightweight second-order access to h_{t-2} through diagonal terms. On TinyShakespeare character-level language modeling, baseVultra achieves a validation loss of 1.594 in the compact one-layer 200K setting, compared with 1.604 for a parameter-matched LSTM and 1.616 for a parameter-matched GRU, while its dense recurrent matrix-vector computation is about 54% of GRU under matched hidden size for this configuration. On text8, a roughly 45x larger benchmark, baseVultra is competitive with full-gate cells (1.854 bpc vs. 1.849 for GRU and 1.837 for LSTM). On formal-language probes, the largest gains appear under constrained hidden-state capacity (0.991 vs. 0.761 GRU and 0.691 LSTM OOD2 accuracy on Counter at H=16) and vanish as hidden size grows. These results characterize where and when candidate-focused recurrent capacity allocation helps: the benefit is concentrated in compact regimes, while full-gate cells remain competitive at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑