利用数据重叠和Prime批量大小减轻序列信息丢失
Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes
摘要：
在序列建模任务中，令牌顺序很重要，但由于序列离散化为数据点，这些信息可能会部分丢失。在本文中，我们研究了某些标记对被包括在数据点和其他人没有的方式之间的不平衡。我们表示这是一个令牌顺序不平衡（TOI），我们链接的部分序列信息的损失作为一个整体，无论是在文本和语音处理任务的系统的性能下降。然后，我们提供了一种机制，通过迭代重叠数据点的代币组成来利用完整的代币订单信息（简化的TOI）。对于递归网络，我们使用素数作为批量大小，以避免在从重叠数据点构建批量时出现冗余。所提出的方法在文本和语音相关的任务中实现了最先进的性能。
英文摘要：
In sequence modeling tasks the token order matters, but this information can be partially lost due to the discretization of the sequence into data points. In this paper, we study the imbalance between the way certain token pairs are included in data points and others are not. We denote this a token order imbalance (TOI) and we link the partial sequence information loss to a diminished performance of the system as a whole, both in text and speech processing tasks. We then provide a mechanism to leverage the full token order information -Alleviated TOI- by iteratively overlapping the token composition of data points. For recurrent networks, we use prime numbers for the batch size to avoid redundancies when building batches from overlapped data points. The proposed method achieved state of the art performance in both text and speech related tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
