MinMax 循环神经网络级联
MinMax Recurrent Neural Cascades
AI总结：
MinMax RNCs 通过MinMax代数构建，具备强表达性、高效评估、稳定动态和非消失状态梯度等特性，在合成任务中表现优异，能处理长序列并超越传统循环基线。
摘要：
我们引入MinMax循环神经网络级联（MinMax RNCs），一种基于MinMax代数新形式递归的循环神经网络。我们展示了MinMax RNCs具有一些难以同时获得的关键性质：强大的形式表达性、高效的评估、稳定的动态和非消失的状态梯度。首先，其形式表达性对应正则语言，可能是有限记忆系统的最大表达性。其次，除了递归形式的评估外，它们还允许并行扫描评估，具有对数深度和线性工作量。第三，其状态和激活在所有序列长度下均被统一限制。第四，其损失梯度几乎处处存在且在所有序列长度下均被统一限制。第五，它们不表现出消失的状态梯度：状态相对于过去状态的梯度可以独立于状态之间的时距保持范数一。经验上，我们发现这些理论性质转化为强大的实际性能。MinMax RNCs完美解决了考虑的合成任务，能够泛化到长序列，并在实验中超越了考虑的循环基线。我们还训练了一个1.12亿参数的MinMax RNC进行下一个token预测，获得与其规模相竞争的性能，提供了初始证据表明MinMax递归可以扩展到现实世界的序列建模任务。
英文摘要：
We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra. We show that MinMax RNCs enjoy key properties that are difficult to obtain simultaneously: strong formal expressivity, efficient evaluation, stable dynamics, and non-vanishing state gradients. First, their formal expressivity corresponds to the regular languages, arguably the maximal expressivity for finite-memory systems. Second, in addition to evaluation in recurrent form, they also admit parallel-scan evaluation with logarithmic depth and linear work in the input length. Third, their states and activations are uniformly bounded for all sequence lengths. Fourth, their loss gradients exist almost everywhere and are uniformly bounded for all sequence lengths. Fifth, they do not exhibit vanishing state gradients: the gradient of a state with respect to a past state can retain norm one independently of the temporal distance between the states. Empirically, we find that these theoretical properties translate into strong practical performance. MinMax RNCs solve the considered synthetic tasks perfectly, generalise to long sequences, and outperform the recurrent baselines considered in our experiments. We also train a 112M-parameter MinMax RNC for next-token prediction, obtaining competitive performance for its size and providing initial evidence that MinMax recurrence can scale to real-world sequence-modelling tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
