Many tasks within NLP can be framed as sequential decision problems, ranging from sequence tagging to text generation. However, for many tasks, the standard training methods, including maximum likelihood (teacher forcing) and scheduled sampling, suffer from exposure bias and a mismatch between metrics employed during training and inference. DAgger provides a solution to mitigate these problems, yet it requires a metric-specific dynamic oracle algorithm, which does not exist for many common metrics like span-based F1, ROUGE, and BLEU. In this paper, we develop these novel dynamic oracles and show they maintain DAgger's no-regret guarantee for decomposable metrics like span-based F1. We evaluate the algorithm's performance on named entity recognition (NER), text summarization, and machine translation (MT). While DAgger with dynamic oracle yields less favorable results in our MT experiments, it outperforms the baseline techniques in NER and text summarization.
Step-unrolled Denoising Autoencoders for Text Generation
DeepMind(深度思维)
在本文中,我们提出了一种新的文本生成模型,步进展开去噪自编码器(SUNDAE),它不依赖于自回归模型。与去噪扩散技术类似,SUNDAE重复应用于一系列令牌,从随机输入开始,每次改进,直到收敛。我们提出了一个简单的新的改进算子,收敛在更少的迭代比扩散方法,同时定性产生更好的样本自然语言数据集。SUNDAE在WMT'14英语到德语翻译任务上取得了最先进的结果(在非自回归方法中),在Colossal Cleaned Common Crawl数据集和GitHub的Python代码数据集上取得了良好的无条件语言建模定性结果。SUNDAE的非自回归性质通过在模板中填充任意空白模式,打开了从左到右提示生成之外的可能性。
Improving Transfer Learning for Sequence Labeling Tasks by Adapting Pre-trained Neural Language Models
FACULTY OF ELECTRICAL ENGINEERING AND COMPUTING(电气工程与计算学院);University of Zagreb, Faculty of Electrical Engineering and Computing(Zagreb大学电气工程与计算学院);Department of Electronics, Microelectronics, Computer and Intelligent systems(电子、微电子、计算机和智能系统系);Text Analysis and Knowledge Engineering Laboratory (TakeLab)(文本分析与知识工程实验室)