arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.07364cs.CL
收藏

迈向增量Transformer:面向增量自然语言理解的Transformer模型实证分析

Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU

University of Potsdam(波茨坦大学);Computational Linguistics(计算语言学(学院/系))

Patrick Kahardipraja, Brielen Madureira, David Schlangen

AI总结:

本文实证分析线性Transformer用于英语增量自然语言理解的可行性,发现循环式LT在增量性能和推理速度上优于标准Transformer及重启增量式LT,并提出等待右侧上下文和输入前缀训练可缓解全序列质量下降。

摘要:

增量处理允许交互系统基于部分输入作出响应,这在对话智能体等场景中是一种理想特性。当前流行的Transformer架构本质上将序列作为整体处理,抽象掉了时间概念。近期工作尝试通过重启增量方式将Transformer应用于增量场景,即反复向未修改的模型输入越来越长的输入前缀以产生部分输出。然而,这种方法计算代价高昂,且无法高效扩展到长序列。与此同时,我们看到使Transformer更高效的努力,例如具有循环机制的线性Transformer(LT)。在本工作中,我们考察了LT用于英语增量自然语言理解的可行性。我们的结果表明,与标准Transformer以及采用重启增量方式的LT相比,循环式LT模型具有更好的增量性能和更快的推理速度,代价是损失部分非增量(全序列)质量。我们表明,可以通过训练模型在提交输出前等待右侧上下文来缓解性能下降,并且使用输入前缀进行训练有助于产生正确的部分输出。

英文摘要:

Incremental processing allows interactive systems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents. The currently popular Transformer architecture inherently processes sequences as a whole, abstracting away the notion of time. Recent work attempts to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. However, this approach is computationally costly and does not scale efficiently for long sequences. In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a recurrence mechanism. In this work, we examine the feasibility of LT for incremental NLU in English. Our results show that the recurrent LT model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental (full sequence) quality. We show that the performance drop can be mitigated by training the model to wait for right context before committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑