Event temporal graphs have been shown as convenient and effective representations of complex temporal relations between events in text. Recent studies, which employ pre-trained language models to auto-regressively generate linearised graphs for constructing event temporal graphs, have shown promising results. However, these methods have often led to suboptimal graph generation as the linearised graphs exhibit set characteristics which are instead treated sequentially by language models. This discrepancy stems from the conventional text generation objectives, leading to erroneous penalisation of correct predictions caused by the misalignment of elements in target sequences. To address these challenges, we reframe the task as a conditional set generation problem, proposing a Set-aligning Framework tailored for the effective utilisation of Large Language Models (LLMs). The framework incorporates data augmentations and set-property regularisations designed to alleviate text generation loss penalties associated with the linearised graph edge sequences, thus encouraging the generation of more relation edges. Experimental results show that our framework surpasses existing baselines for event temporal graph generation. Furthermore, under zero-shot settings, the structural knowledge introduced through our framework notably improves model generalisation, particularly when the training examples available are limited.
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
ElementAI;ServiceNow
用于文本数据的大型预训练语言模型具有不受约束的输出空间;在每个解码步骤中,它们可以产生10,000个子单词标记中的任何一个。当对目标约束形式语言(如SQL)进行微调时,这些模型通常会生成无效代码,从而使其无法使用。我们提出了PICARD(code and trained models available at https://github.com/ElementAI/picard),这是一种通过增量解析来约束语言模型的自回归解码器的方法。PICARD通过在每个解码步骤中拒绝不可接受的令牌来帮助找到有效的输出序列。在具有挑战性的Spider和CoSQL文本到SQL的翻译任务中,我们展示了PICARD将性能尚可的微调T5模型转换为最先进的解决方案。
Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education(高可信软件技术重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Electronic Science and Technology of China(电子科技大学)
Lossless Vocabulary Reduction for Auto-Regressive Language Models
NTT Computer and Data Science Laboratories, NTT Corporation(NTT计算机与数据科学实验室,NTT集团);NTT Human Informatics Laboratories, NTT Corporation(NTT人类信息学实验室,NTT集团)
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Digital Trust Centre, Nanyang Technological University(南洋理工大学数字信托中心);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models
Alibaba-NTU Global e-Sustainability CorpLab(阿里-国立大学全球可持续发展公司实验室);Nanyang Technological University(南洋理工大学);College of Computing and Data Science(计算与数据科学学院);Alibaba(阿里);Alibaba Inc.(阿里公司)