Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
Lecheng Wang, Xianjie Shi, Ge Li, Jia Li, Xuanming Zhang, Yihong Dong, Wenpin Jiao, Hong Mei
Auto-regressive language models (LMs) have been widely used to generate data in data-scarce domains to train new LMs, compensating for the scarcity of real-world data. Previous work experimentally found that LMs collapse when trained on recursively generated data. This paper presents a theoretical proof: once a corpus (such as a subset of the World Wide Web) begins to incorporate generated data and no new real-world data is added to the corpus, then no matter how small the amount of data each LM generates and contributes to the corpus, LM collapse is inevitable after sufficient time. This finding suggests that attempts to mitigate collapse by limiting the quantity of synthetic data in the corpus are fundamentally insufficient. Instead, avoiding collapse hinges on ensuring the quality of synthetic data.
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Electronic Science and Technology of China(电子科技大学)
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
ElementAI;ServiceNow
用于文本数据的大型预训练语言模型具有不受约束的输出空间;在每个解码步骤中,它们可以产生10,000个子单词标记中的任何一个。当对目标约束形式语言(如SQL)进行微调时,这些模型通常会生成无效代码,从而使其无法使用。我们提出了PICARD(code and trained models available at https://github.com/ElementAI/picard),这是一种通过增量解析来约束语言模型的自回归解码器的方法。PICARD通过在每个解码步骤中拒绝不可接受的令牌来帮助找到有效的输出序列。在具有挑战性的Spider和CoSQL文本到SQL的翻译任务中,我们展示了PICARD将性能尚可的微调T5模型转换为最先进的解决方案。
Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models
Alibaba-NTU Global e-Sustainability CorpLab(阿里-国立大学全球可持续发展公司实验室);Nanyang Technological University(南洋理工大学);College of Computing and Data Science(计算与数据科学学院);Alibaba(阿里);Alibaba Inc.(阿里公司)
Lossless Vocabulary Reduction for Auto-Regressive Language Models
NTT Computer and Data Science Laboratories, NTT Corporation(NTT计算机与数据科学实验室,NTT集团);NTT Human Informatics Laboratories, NTT Corporation(NTT人类信息学实验室,NTT集团)
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Digital Trust Centre, Nanyang Technological University(南洋理工大学数字信托中心);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)