Large Language Models (LLMs) have demonstrated remarkable performance across diverse natural language processing tasks, yet their ability to memorize structured knowledge remains underexplored. In this paper, we investigate the extent to which general-purpose pre-trained LLMs retain and correctly reproduce concept identifier (ID)-label associations from publicly available ontologies. We conduct a systematic evaluation across multiple ontological resources, including the Gene Ontology, Uberon, Wikidata, and ICD-10, using LLMs such as Pythia-12B, Gemini-1.5-Flash, GPT-3.5, and GPT-4. Our findings reveal that only a small fraction of ontological concepts is accurately memorized, with GPT-4 demonstrating the highest performance. To understand why certain concepts are memorized more effectively than others, we analyze the relationship between memorization accuracy and concept popularity on the Web. Our results indicate a strong correlation between the frequency of a concept's occurrence online and the likelihood of accurately retrieving its ID from the label. This suggests that LLMs primarily acquire such knowledge through indirect textual exposure rather than directly from structured ontological resources. Furthermore, we introduce new metrics to quantify prediction invariance, demonstrating that the stability of model responses across variations in prompt language and temperature settings can serve as a proxy for estimating memorization robustness.
Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers Evidence Across Models and Ontologies
Department of Neurology(神经学系);Rehabilitation University of Illinois at Chicago Chicago, IL, USA(伊利诺伊大学芝加哥分校康复学院);Lab for Applied Artificial Intelligence(应用人工智能实验室);Engineering Program(工程学院)
Ontology Learning with LLMs: A Benchmark Study on Axiom Identification
Netherlands Organisation for Applied Scientific Research (TNO)(荷兰应用科学研究院);Leiden University Centre for Linguistics (LUCL)(莱顿大学语言研究中心);Utrecht University(乌得勒支大学)
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);School of Foreign Languages, East China University of Science and Technology(华东理工大学外国语学院);Mejiro University(目白大学)
Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
University of California Irvine(加州大学欧文分校);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Paul G. Allen School of Computer Science(保罗·G·艾伦计算机科学学院)
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
Courant Institute of Mathematical Sciences, New York University(纽约大学数学科学学院);Tandon School of Engineering, New York University(纽约大学工程学院);Cerebras Systems Inc.(Cerebras Systems公司);University of Pennsylvania(宾夕法尼亚大学)
Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study
Knowledge Media Institute, The Open University(开放大学知识媒体学院);The Open University(开放大学);University of Milano Bicocca(米兰比克卡大学);Department of Business and Law, University of Milano Bicocca(米兰比克卡大学商学院与法学院)
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Mila & Université de Montréal(Mila与蒙特利尔大学);Noah’s Ark Lab(Noah’s Ark实验室);Advanced Micro Devices;Chandar Research Lab(Chandar研究实验室);Polytechnique Montréal(蒙特利尔理工学院);CIFAR AI Chair(CIFAR人工智能主席)
Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field
Knowledge Media Institute, The Open University, Milton Keynes, UK(开放大学知识媒体研究所);Department of Business and Law, University of Milano-Bicocca, Milan, IT(米兰-比科卡大学商业与法律系)
LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Models
TIB Leibniz Information Centre for Science and Technology(莱布尼茨科学与技术信息中心(TIB));TU/e Eindhoven University of Technology(埃因霍温理工大学);University of Warwick(华威大学);University of Brescia(布雷西亚大学);L3S Research Center(L3S研究中心)