用于持续学习的Minion门控循环单元
Minion Gated Recurrent Unit for Continual Learning
AI总结：
本文提出简化版GRU变体MiRU，用缩放系数替代门控机制，在保持性能的同时实现2.90倍训练加速和2.88倍参数缩减，并展现出更稳定的多任务持续学习能力，适合边缘设备部署。
摘要：
顺序数据处理中对持续学习日益增长的需求导致了越来越复杂的训练方法和更大的循环网络架构。因此，这扩大了基于循环神经网络（RNNs）的持续学习与其在内存和计算资源有限的设备上运行能力之间的知识差距。为了应对这一挑战，我们研究了简化RNN架构（特别是门控循环单元（GRU））的有效性，及其对单任务和多任务顺序学习的影响。我们提出了一种新的GRU变体，即minion循环单元（MiRU）。MiRU用缩放系数替代传统的门控机制，以调节隐藏状态和历史上下文的动态更新，从而降低计算成本和内存需求。尽管架构简化，MiRU仍保持与标准GRU相当的性能，同时实现了2.90倍的训练加速，并将参数使用量减少了2.88倍，这一点通过在顺序图像分类和自然语言处理基准上的评估得到了证明。我们还通过采用基于回放的策略和全局抑制执行持续学习任务，研究了模型简化对其学习能力的影响。我们发现，与标准GRU及其变体不同，MiRU即使仅使用回放也能在多任务学习中表现出稳定的性能。这些特性使MiRU成为边缘设备应用的一个有前景的候选方案。
英文摘要：
The increasing demand for continual learning in sequential data processing has led to progressively complex training methodologies and larger recurrent network architectures. Consequently, this has widened the knowledge gap between continual learning with recurrent neural networks (RNNs) and their ability to operate on devices with limited memory and compute. To address this challenge, we investigate the effectiveness of simplifying RNN architectures, particularly gated recurrent unit (GRU), and its impact on both single-task and multitask sequential learning. We propose a new variant of GRU, namely the minion recurrent unit (MiRU). MiRU replaces conventional gating mechanisms with scaling coefficients to regulate dynamic updates of hidden states and historical context, reducing computational costs and memory requirements. Despite its simplified architecture, MiRU maintains performance comparable to the standard GRU while achieving 2.90x faster training and reducing parameter usage by 2.88x, as demonstrated through evaluations on sequential image classification and natural language processing benchmarks. The impact of model simplification on its learning capacity is also investigated by performing continual learning tasks with a rehearsal-based strategy and global inhibition. We find that MiRU demonstrates stable performance in multitask learning even when using only rehearsal, unlike the standard GRU and its variants. These features position MiRU as a promising candidate for edge-device applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
