We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding.
FullStop:Punctuation and Segmentation Prediction for Dutch with Transformers
Instituut voor de Nederlandse Taal（荷兰语言研究所）；KU Leuven（鲁汶大学）；Hochschule für Technik und Wirtschaft（德累斯顿工程与经济应用科学大学）
当对比利时荷兰语应用自动语音识别（ASR）时（Van Dyck et al. 2021），输出由未分段的单词流组成，没有任何标点符号。下一步是执行分段和插入标点符号，使ASR输出更具可读性，易于手动更正。据我们所知，没有一个公共可用的标点符号插入系统的荷兰语，功能在一个可用的水平。我们在这里提出的模型是Guhr等人（2021）荷兰模型的扩展，并公开提供。我们基于荷兰语言模型RobBERT（Delobelle et al. 2020）训练了一个序列分类模型。对于输入序列中的每个单词，模型都会预测该单词后面的标点符号。我们还扩展了多语言模型，用于语言未知或代码切换适用的情况。当执行分割任务时，将最佳模型应用到域外测试数据上，将ASR输出流的200个单词的滑动窗口发送到分类器，并且当系统预测具有高于阈值的比率的分割标点符号时应用分割。结果表明，它比机器翻译基线方法要好得多。
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
ETH Zurich（苏黎世联邦理工学院）；Allen Institute for AI（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Tractable Transformers for Flexible Conditional Generation
Department of Computer Science, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学校）
