关注双向序列生成的未来令牌
Attending to Future Tokens For Bidirectional Sequence Generation
摘要:
神经序列生成通常是逐令牌和从左到右执行的。每当生成令牌时,只考虑先前生成的令牌。相比之下,对于序列分类等问题,双向注意力,即考虑过去和未来的令牌,已被证明表现得更好。我们建议通过使用特殊的占位符标记使序列生成过程是双向的。作为全连接图中的节点,占位符标记可以在生成实际输出标记时考虑过去和未来的标记。我们验证了我们的方法的有效性实验上的两个会话任务,建议的双向模型优于竞争基线的大幅度。
英文摘要:
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversational tasks where the proposed bidirectional model outperforms competitive baselines by a large margin.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。