Conceptual knowledge is fundamental to human cognition and knowledge bases. However, existing knowledge probing works only focus on evaluating factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge. Since conceptual knowledge often appears as implicit commonsense behind texts, designing probes for conceptual knowledge is hard. Inspired by knowledge representation schemata, we comprehensively evaluate conceptual knowledge of PLMs by designing three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts, respectively. For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark. Extensive experiments on different sizes and types of PLMs show that existing PLMs systematically lack conceptual knowledge and suffer from various spurious correlations. We believe this is a critical bottleneck for realizing human-like cognition in PLMs. COPEN and our codes are publicly released at https://github.com/THU-KEG/COPEN.
Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；BenchCouncil (International Open Benchmark Council)（Benchmark委员会）；University of Chinese Academy of Sciences（中国科学院大学）；School of Artificial Intelligence and Data Science, Hebei University of Technology（河北工业大学人工智能与数据科学学院）
On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究院）；Department of Psychology, Harvard University（哈佛大学心理学系）
Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs
Saarland University（萨尔大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Centre for European Research in Trusted AI (CERTAIN)（欧洲可信人工智能研究中心）；University of Göttingen（哥廷根大学）
Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?
Rutgers University（罗格斯大学）；University of Exeter（埃克塞特大学）；Northwestern University（西北大学）；New Jersey Institute of Technology（新泽西理工学院）；Wake Forest University（维克森林大学）
