arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.11603cs.CL
收藏

用循环神经网络消除序列预测中的暴露偏差

Alleviate Exposure Bias in Sequence Prediction \\ with Recurrent Neural Networks

Fudan University(复旦大学)

Liping Yuan, Jiangtao Feng, Xiaoqing Zheng, Xuanjing Huang

摘要:

训练循环神经网络(RNN)的一种流行策略,称为"教师强迫“,在每个时间步将地面事实作为输入,并使后来的预测部分取决于这些输入。这种训练策略削弱了它们在整个序列上学习丰富分布的能力,因为所选择的输入阻碍了梯度以端到端的方式反向传播到所有先前的状态。我们提出了一种完全可微的RNN训练算法,通过恢复整个序列的概率来更好地捕获长期依赖关系。其关键思想是,在每个时间步,网络将在前一步预测的类似单词的“集合”作为输入,而不是单一的基础事实。这些相似词的表示形成了一个凸包,它可以被看作是对输入的一种正则化。通过这种方式平滑输入使得整个过程可训练和可微分。这种设计使模型能够探索更多可行的组合(可能是看不见的序列),并且可以被解释为对束搜索的计算高效近似。多个序列生成任务的实验产生的性能改进,特别是在序列级的指标,如蓝色或ROUGE-2。

英文摘要:

A popular strategy to train recurrent neural networks (RNNs), known as ``teacher forcing'' takes the ground truth as input at each time step and makes the later predictions partly conditioned on those inputs. Such training strategy impairs their ability to learn rich distributions over entire sequences because the chosen inputs hinders the gradients back-propagating to all previous states in an end-to-end manner. We propose a fully differentiable training algorithm for RNNs to better capture long-term dependencies by recovering the probability of the whole sequence. The key idea is that at each time step, the network takes as input a ``bundle'' of similar words predicted at the previous step instead of a single ground truth. The representations of these similar words forms a convex hull, which can be taken as a kind of regularization to the input. Smoothing the inputs by this way makes the whole process trainable and differentiable. This design makes it possible for the model to explore more feasible combinations (possibly unseen sequences), and can be interpreted as a computationally efficient approximation to the beam search. Experiments on multiple sequence generation tasks yield performance improvements, especially in sequence-level metrics, such as BLUE or ROUGE-2.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑