arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.00236cs.LGcs.NE
收藏

位置编码助力循环神经网络处理大词汇量

Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary

Academy of Emerging Sciences, Chubu University(中部大学新兴科学学院);Center for Mathematical Science and AI, Chubu University(中部大学数学与人工智能中心)

Takashi Morita

AI总结:

本研究发现位置编码可增强循环神经网络(RNNs)的学习能力,尤其在处理含低频标记的大词汇量时,能解决普通RNNs因低频标记导致的梯度不稳定问题,拓展了位置编码的应用场景。

摘要:

本研究报告了一个反直觉的发现:位置编码可增强循环神经网络(RNNs)的学习能力。位置编码是输入数据时间索引的高维表示,最著名的是它能补充Transformer神经网络的能力,后者缺乏表示数据顺序的固有机制。相比之下,RNNs可自行编码数据点的时间信息,这使得位置编码的使用看似冗余或不必要。然而,通过合成基准测试的研究表明,结合位置编码与RNNs存在优势,尤其在处理包含低频标记的大词汇量时。进一步分析揭示,这些低频标记会破坏普通RNNs的梯度稳定性,而位置编码可解决这一不稳定性。这些结果为位置编码在Transformer的典型计时角色之外的效用提供了新视角。

英文摘要:

This study reports an unintuitive finding that positional encoding enhances learning of recurrent neural networks (RNNs). Positional encoding is a high-dimensional representation of time indices on input data. Most famously, positional encoding complements the capabilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly redundant/unnecessary. Nonetheless, investigations through synthetic benchmarks reveal an advantage of coupling positional encoding and RNNs, especially for handling a large vocabulary that yields low-frequency tokens. Further scrutinization unveils that these low-frequency tokens destabilizes the gradients of vanilla RNNs, and the positional encoding resolves this instability. These results shed a new light on the utility of positional encoding beyond its canonical role as a timekeeper for Transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑