用于条件序列处理的聚焦层次RNN
Focused Hierarchical RNNs for Conditional Sequence Processing
摘要：
具有注意力机制的递归神经网络（RNN）已经在许多序列处理任务中获得了最先进的结果。这些模型中的大多数使用一种简单形式的编码器，它会查看整个序列并独立地为每个令牌分配权重。我们提出了一种将RNN编码器集中用于序列建模任务的机制，允许它们根据需要处理输入的关键部分。我们使用多层条件序列编码器来制定这一点，该编码器每次读取一个令牌，并对令牌是否与上下文或所问问题相关做出离散决策。离散门控机制将上下文嵌入和当前隐藏状态作为输入，并控制信息流进入上一层。我们使用策略梯度方法对其进行训练。我们评估这种方法上的几种类型的任务具有不同的属性。首先，我们在合成任务上评估该方法，这使我们能够评估模型的泛化能力，并在更受控的设置中探测门的行为。然后，我们评估这种方法在大规模的问题分类任务，包括具有挑战性的MS MARCO和SearchQA任务。我们的模型显示，这两项任务都比以前的工作和我们的基线有了一致的改进。与基线相比，它在合成任务上的泛化效果也明显更好。
英文摘要：
Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing RNN encoders for sequence modelling tasks which allows them to attend to key parts of the input as needed. We formulate this using a multi-layer conditional sequence encoder that reads in one token at a time and makes a discrete decision on whether the token is relevant to the context or question being asked. The discrete gating mechanism takes in the context embedding and the current hidden state as inputs and controls information flow into the layer above. We train it using policy gradient methods. We evaluate this method on several types of tasks with different attributes. First, we evaluate the method on synthetic tasks which allow us to evaluate the model for its generalization ability and probe the behavior of the gates in more controlled settings. We then evaluate this approach on large scale Question Answering tasks including the challenging MS MARCO and SearchQA tasks. Our models shows consistent improvements for both tasks over prior work and our baselines. It has also shown to generalize significantly better on synthetic tasks as compared to the baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
