With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge. The mere pre-trained language models, however, lack the capacity of handling such knowledge-intensive NLP tasks alone. To address this challenge, large numbers of pre-trained language models augmented with external knowledge sources are proposed and in rapid development. In this paper, we aim to summarize the current progress of pre-trained language model-based knowledge-enhanced models (PLMKEs) by dissecting their three vital elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods. Finally, we present the challenges of PLMKEs based on the discussion regarding the three elements and attempt to provide NLP practitioners with potential directions for further research.
Enhancing NLP Robustness and Generalization through LLM-Generated Contrast Sets: A Scalable Framework for Systematic Evaluation and Adversarial Training
College of Natural Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校自然科学学院）
Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model
Weill Cornell Medicine（威尔康奈尔医学院）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Mayo Clinic（梅奥诊所）；Columbia University Vagelos College of Physicians and Surgeons（哥伦比亚大学瓦格洛斯内科医生与外科医生学院）；New York State Psychiatric Institute（纽约州精神病学研究所）
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus
Facebook AI Research（脸书人工智能研究院）；University College London（伦敦大学学院）；University of Mannheim（曼海姆大学）；ENS, PSL University（巴黎高等师范学院，巴黎文理研究大学）；Inria（法国国家信息与自动化研究所）；University of Washington（华盛顿大学）
From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding
Department of Electrical and Computer Engineering, Tufts University（电气与计算机工程系，塔夫茨大学）；Department of Computer Science and Information Technology, Patuakhali Science and Technology University（计算机科学与信息技术系，帕图阿哈利科学与技术大学）；Department of Electrical and Computer Engineering, Computer Science and Biomedical Engineering, Tufts University（电气与计算机工程系，计算机科学与生物医学工程系，塔夫茨大学）；Department of Mechanical Engineering and Education, Tufts University（机械工程与教育系，塔夫茨大学）
Adapting Natural Language Processing Models Across Jurisdictions: A pilot Study in Canadian Cancer Registries
School of Population and Public Health University of British Columbia（流行病学与公共卫生学院 首都大学）；Newfoundland & Labrador Health Services（纽芬兰与拉布拉多省卫生服务）；Data Science Institute University of British Columbia（数据科学研究所 首都大学）
NLPBench: Evaluating Large Language Models on Solving NLP Problems
Waseda University（早稻田大学）；University of Tokyo（东京大学）；University of Washington（华盛顿大学）；Zhejiang Lab（之江实验室）；University of Science and Technology of China（中国科学技术大学）；University of Maryland（马里兰大学）
Can LLM-Generated Textual Explanations Enhance Model Classification Performance? An Empirical Study
Technical University of Munich, School of Computation, Information and Technology, Department of Computer Science, Munich, Germany（慕尼黑技术大学，计算、信息与技术学院，计算机科学系，德国慕尼黑）
Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
School of Computational Science \& Engineering Georgia Institute of Technology Atlanta GA USA；Scheller College of Business Georgia Institute of Technology Atlanta GA USA；Georgia Institute of Technology
Knowledge Rumination for Pre-trained Language Models
Zhejiang University（浙江大学）；Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）；Alibaba Group（阿里巴巴集团）；Donghai Laboratory（东海实验室）
Developing and Evaluating an AI-Assisted Prediction Model for Unplanned Intensive Care Admissions following Elective Neurosurgery using Natural Language Processing within an Electronic Healthcare Record System
University College London（伦敦大学学院）；National Hospital for Neurology and Neurosurgery（国立神经学与神经外科医院）
On the Effectiveness of Compact Biomedical Transformers
NLPie（NLPie公司）
在生物医学语料库上预训练的语言模型，如BioBERT，最近在下游生物医学任务上显示出有希望的结果。另一方面，由于嵌入大小、隐藏维度和层数等因素，许多现有的预训练模型都是资源密集型的，计算量很大。自然语言处理（NLP）社区已经开发了许多策略来压缩这些模型，使用修剪，量化和知识蒸馏等技术，从而产生更快，更小，更容易在实践中使用的模型。出于同样的原因，在本文中，我们介绍了六个轻量级模型，即BioDistilBERT，BioTinyBERT，BioMobileBERT，DistilBioBERT，TinyBioBERT和CompactBioBERT，它们是通过生物医学教师的知识蒸馏或通过Masked Language Modeling（MLM）目标在Pubmed数据集上进行持续学习而获得的。我们在三个生物医学任务上评估了我们所有的模型，并将它们与BioBERT-v1.1进行比较，以创建与大型模型不相上下的高效轻量级模型。所有的模型都将在我们的Huggingface个人资料中公开，网址是https://huggingface.co/nlpie，用于运行实验的代码将在https://github.com/nlpie-research/Compact-Biomedical-Transformers上提供。
Eeny, meeny, miny, moe. How to choose data for morphological inflection
The Australian National University (ANU)（澳大利亚国立大学）；University of Colorado（科罗拉多大学）；ARC Centre of Excellence for the Dynamics of Language (CoEDL)（澳大利亚研究理事会语言动态卓越中心）
The Nature of NLP: Analyzing Contributions in NLP Papers
Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technische Universität Darmstadt（德累斯顿技术大学计算机科学系及海瑟森人工智能中心）；IT:U Interdisciplinary Transformation University Austria（奥地利 interdisciplinary Transformation 大学）；IBM Research Europe - Ireland（IBM欧洲爱尔兰研究中心）；National Research Council Canada（加拿大国家研究委员会）
