A Generative Approach for Script Event Prediction via Contrastive Fine-tuning
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Beijing University of Technology（北京工业大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Fangqi Zhu, Jun Gao, Changlong Yu, Wei Wang, Chen Xu, Xin Mu, Min Yang, Ruifeng Xu
Script event prediction aims to predict the subsequent event given the context. This requires the capability to infer the correlations between events. Recent works have attempted to improve event correlation reasoning by using pretrained language models and incorporating external knowledge~(e.g., discourse relations). Though promising results have been achieved, some challenges still remain. First, the pretrained language models adopted by current works ignore event-level knowledge, resulting in an inability to capture the correlations between events well. Second, modeling correlations between events with discourse relations is limited because it can only capture explicit correlations between events with discourse markers, and cannot capture many implicit correlations. To this end, we propose a novel generative approach for this task, in which a pretrained language model is fine-tuned with an event-centric pretraining objective and predicts the next event within a generative paradigm. Specifically, we first introduce a novel event-level blank infilling strategy as the learning objective to inject event-level knowledge into the pretrained language model, and then design a likelihood-based contrastive loss for fine-tuning the generative model. Instead of using an additional prediction layer, we perform prediction by using sequence likelihoods generated by the generative model. Our approach models correlations between events in a soft way without any external knowledge. The likelihood-based prediction eliminates the need to use additional networks to make predictions and is somewhat interpretable since it scores each word in the event. Experimental results on the multi-choice narrative cloze~(MCNC) task demonstrate that our approach achieves better results than other state-of-the-art baselines. Our code will be available at https://github.com/zhufq00/mcnc.
Multi-level Contrastive Learning for Script-based Character Understanding
Halicioğlu Data Science Institute, University of California, San Diego（加州大学圣地亚哥分校哈利乔奥卢数据科学研究所）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）
预先训练的语言模型已经在广泛的自然语言处理（NLP）任务中取得了实质性的进展，但已经被证明对于具有高质量输出要求的自然语言生成任务具有局限性，例如常识生成和广告关键字生成。在这项工作中，我们提出了一种新的知识过滤和对比学习网络（KFCNet），参考外部知识，并取得更好的生成性能。具体来说，我们提出了一个基于BERT的过滤器模型，以消除低质量的候选人，并分别应用对比学习的编码器和解码器中的每一个，在一个通用的编码器-解码器架构。编码器对比模块有助于在编码过程中捕获全局目标语义，解码器对比模块在学习一般特征的同时增强检索到的原型的实用性。在CommonGen基准测试上进行的大量实验表明，我们的模型远远优于先前的最先进水平：BLEU-4为+6.6点（42.5 vs. 35.9），SPICE为+3.7点（33.3 vs. 29.6），CIDER为+1.3点（18.3 vs. 17.0）。我们进一步验证了所提出的对比模块在广告关键词生成上的有效性，并表明我们的模型具有潜在的商业价值。
Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement
Department of Computer Science and Information Engineering, National Taiwan University（国家台湾大学计算机科学与信息工程系）；National Taiwan University AI Center of Research Excellence（国家台湾大学人工智能研究中心）
GRACE: Generative Representation Learning via Contrastive Policy Optimization
University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）；Australian National University（澳大利亚国立大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of Washington（华盛顿大学）
Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition
Univ Toulon, Aix Marseille Univ, LIS, CNRS（土伦大学，艾克斯-马赛大学，LIS，法国国家科学研究中心）；Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；Idiap Research Institute（Idiap 研究所）；Sapienza University of Rome（罗马大学）；Brno University of Technology（布尔诺理工大学）；pyannoteAI（pyannoteAI 公司）；Concordia University（康考迪亚大学）；CNRS, ILLS（法国国家科学研究中心，ILLS）
CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scripts
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Amazon（亚马逊）；Salesforce Research（Salesforce 研究院）；Harrisburg University of Science and Technology（哈里斯堡科技大学）
