Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
ElementAI；ServiceNow
用于文本数据的大型预训练语言模型具有不受约束的输出空间;在每个解码步骤中，它们可以产生10，000个子单词标记中的任何一个。当对目标约束形式语言（如SQL）进行微调时，这些模型通常会生成无效代码，从而使其无法使用。我们提出了PICARD（code and trained models available at https：//github.com/ElementAI/picard），这是一种通过增量解析来约束语言模型的自回归解码器的方法。PICARD通过在每个解码步骤中拒绝不可接受的令牌来帮助找到有效的输出序列。在具有挑战性的Spider和CoSQL文本到SQL的翻译任务中，我们展示了PICARD将性能尚可的微调T5模型转换为最先进的解决方案。
Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术重点实验室（北京大学））；School of Computer Science, Peking University（北京大学计算机学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University of Electronic Science and Technology of China（电子科技大学）
Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models
Alibaba-NTU Global e-Sustainability CorpLab（阿里-国立大学全球可持续发展公司实验室）；Nanyang Technological University（南洋理工大学）；College of Computing and Data Science（计算与数据科学学院）；Alibaba（阿里）；Alibaba Inc.（阿里公司）
Lossless Vocabulary Reduction for Auto-Regressive Language Models
NTT Computer and Data Science Laboratories, NTT Corporation（NTT计算机与数据科学实验室，NTT集团）；NTT Human Informatics Laboratories, NTT Corporation（NTT人类信息学实验室，NTT集团）
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Digital Trust Centre, Nanyang Technological University（南洋理工大学数字信托中心）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
