Pre-trained language models (PLMs) have been prevailing in state-of-the-art methods for natural language processing, and knowledge-enhanced PLMs are further proposed to promote model performance in knowledge-intensive tasks. However, conceptual knowledge, one essential kind of knowledge for human cognition, still remains understudied in this line of research. This limits PLMs' performance in scenarios requiring human-like cognition, such as understanding long-tail entities with concepts. In this paper, we propose ConcEPT, which stands for Concept-Enhanced Pre-Training for language models, to infuse conceptual knowledge into PLMs. ConcEPT exploits external taxonomies with entity concept prediction, a novel pre-training objective to predict the concepts of entities mentioned in the pre-training contexts. Unlike previous concept-enhanced methods, ConcEPT can be readily adapted to various downstream applications without entity linking or concept mapping. Results of extensive experiments show the effectiveness of ConcEPT in four tasks such as entity typing, which validates that our model gains improved conceptual knowledge with concept-enhanced pre-training.
AI-enhanced semantic feature norms for 786 concepts
Dept. of Psychology, University of Wisconsin-Madison(威斯康星大学麦迪逊分校心理学系);Dept. of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系);Wisconsin Institute for Discovery(威斯康星发现研究所);Dept. of Psychology, Stanford University(斯坦福大学心理学系);Dept. of Psychology, Princeton University(普林斯顿大学心理学系);Dept. of Computer Science, Brown University(布朗大学计算机科学系);Princeton Neuroscience Institute(普林斯顿神经科学研究所)
Event Ontology Expansion via LLM-Based Conceptualization
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院)
Domain Knowledge-Enhanced LLMs for Fraud and Concept Drift Detection
School of Computing and Augmented Intelligence (SCAI), Arizona State University (ASU)(计算与增强智能学院(SCAI),亚利桑那州立大学);Department of Computer Engineering, Tarsus University(计算机工程系,塔鲁斯大学);Minerva CQ and HumaConn AI Consulting(Minerva CQ和HumaConn人工智能咨询);School of Computing and Augmented Intelligence (SCAI), Arizona State University(计算与增强智能学院(SCAI),亚利桑那州立大学)
Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
The Chinese University of Hong Kong(香港中文大学);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)(深圳市人工智能与机器人研究院);Guangdong-Hongkong-Macao Greater Bay Area Weather Research Center for Monitoring Warning and Forecasting (Shenzhen Institute of Meteorological Innovation)(粤港澳大湾区天气监测预警预报研究中心(深圳市气象创新研究院));Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳高等研究院)
A Pluggable Common Sense-Enhanced Framework for Knowledge Graph Completion
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);College of Information and Communication Engineering, Hainan University(海南大学信息与通信工程学院)
Redefining <Creative> in Dictionary: Towards an Enhanced Semantic Understanding of Creative Generation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(教育部新一代人工智能技术及交叉应用重点实验室(东南大学))