Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.
Lossless Vocabulary Reduction for Auto-Regressive Language Models
NTT Computer and Data Science Laboratories, NTT Corporation(NTT计算机与数据科学实验室,NTT集团);NTT Human Informatics Laboratories, NTT Corporation(NTT人类信息学实验室,NTT集团)
Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Electronic Science and Technology of China(电子科技大学)
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
ElementAI;ServiceNow
用于文本数据的大型预训练语言模型具有不受约束的输出空间;在每个解码步骤中,它们可以产生10,000个子单词标记中的任何一个。当对目标约束形式语言(如SQL)进行微调时,这些模型通常会生成无效代码,从而使其无法使用。我们提出了PICARD(code and trained models available at https://github.com/ElementAI/picard),这是一种通过增量解析来约束语言模型的自回归解码器的方法。PICARD通过在每个解码步骤中拒绝不可接受的令牌来帮助找到有效的输出序列。在具有挑战性的Spider和CoSQL文本到SQL的翻译任务中,我们展示了PICARD将性能尚可的微调T5模型转换为最先进的解决方案。
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Digital Trust Centre, Nanyang Technological University(南洋理工大学数字信托中心);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)