arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1807.08280cs.CLcs.LGcs.SDeess.AS
收藏

Sequence-to-Sequence模型中注意力机制的多尺度比对和上下文历史

Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model

Nara Institute of Science and Technology(奈良先端科学技术大学院大学);RIKEN(理化学研究所)

Andros Tjandra, Sakriani Sakti, Satoshi Nakamura

摘要:

序列到序列模型是用于映射两个不同长度的序列的神经网络模块。序列到序列模型有三个核心模块:编码器、解码器和注意力。注意力是连接编码器和解码器模块的桥梁,并在许多任务中提高模型性能。在本文中,我们提出了两个想法,以提高序列到序列模型的性能,通过增强注意力模块。首先,我们维护位置的历史和来自先前几个时间步的预期上下文。其次,我们应用多尺度卷积从几个以前的注意向量到当前的解码器状态。我们利用我们提出的序列到序列语音识别和文本到语音系统的框架。结果表明,我们提出的扩展可以显着提高性能相比,一个标准的注意力基线。

英文摘要:

A sequence-to-sequence model is a neural network module for mapping two sequences of different lengths. The sequence-to-sequence model has three core modules: encoder, decoder, and attention. Attention is the bridge that connects the encoder and decoder modules and improves model performance in many tasks. In this paper, we propose two ideas to improve sequence-to-sequence model performance by enhancing the attention module. First, we maintain the history of the location and the expected context from several previous time-steps. Second, we apply multiscale convolution from several previous attention vectors to the current decoder state. We utilized our proposed framework for sequence-to-sequence speech recognition and text-to-speech systems. The results reveal that our proposed extension could improve performance significantly compared to a standard attention baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑