通过分组分销商单元学习长期技能
Learning Longer-term Dependencies via Grouped Distributor Unit
摘要:
尽管最近在序列建模方面取得了成功,但递归神经网络(RNN)学习长期依赖关系仍然很困难。在本文中,我们提出了一种新的门RNN结构,它只包含一个门。建议的分组分发器单元(GDU)中的隐藏状态被划分成组。对于每个组,在每个状态转换中要重写的存储器的比例被限制为常数,并且自适应地分配给每个组成员。换句话说,每个单独的组都有一个固定的整体更新速率,但允许所有单元具有不同的速度。因此,信息被迫以灵活的方式锁定,这有助于模型捕获数据中的长期依赖关系。除了具有更简单的结构外,实验证明GDU在包括病理问题和自然数据集在内的任务上优于LSTM和GRU。
英文摘要:
Learning long-term dependencies still remains difficult for recurrent neural networks (RNNs) despite their success in sequence modeling recently. In this paper, we propose a novel gated RNN structure, which contains only one gate. Hidden states in the proposed grouped distributor unit (GDU) are partitioned into groups. For each group, the proportion of memory to be overwritten in each state transition is limited to a constant and is adaptively distributed to each group member. In other word, every separate group has a fixed overall update rate, yet all units are allowed to have different paces. Information is therefore forced to be latched in a flexible way, which helps the model to capture long-term dependencies in data. Besides having a simpler structure, GDU is demonstrated experimentally to outperform LSTM and GRU on tasks including both pathological problems and natural data set.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。