arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.14931cs.CLcs.AI
收藏

LLM会梦见本体吗?

Do LLMs Dream of Ontologies?

University of Verona(维罗纳大学);University of Mannheim(曼海姆大学)

Marco Bombieri, Paolo Fiorini, Simone Paolo Ponzetto, Marco Rospocher

AI总结:

本文评估了通用预训练LLMs对公开本体中概念ID-标签关联的记忆能力,发现仅小部分概念被准确记忆且与网络出现频率强相关,并引入预测不变性指标以响应稳定性估计记忆鲁棒性。

摘要:

大型语言模型(LLMs)在各类自然语言处理任务中表现出色,但其记忆结构化知识的能力仍有待深入研究。本文探讨了通用预训练LLMs在多大程度上能够保留并正确复现来自公开本体的概念标识符(ID)与标签的关联。我们使用Pythia-12B、Gemini-1.5-Flash、GPT-3.5和GPT-4等模型,对包括Gene Ontology、Uberon、Wikidata和ICD-10在内的多种本体资源进行了系统评估。研究发现,仅有小部分本体概念被准确记忆,其中GPT-4表现最佳。为探究某些概念被更有效记忆的原因,我们分析了记忆准确率与概念在网络上流行度之间的关系。结果表明,概念在线上出现的频率与其从标签准确检索其ID的可能性之间存在强相关性。这表明LLMs主要通过间接文本接触获取此类知识,而非直接源自结构化本体资源。此外,我们引入了新指标来量化预测不变性,证明模型在提示语言和温度设置变化下的响应稳定性可作为估计记忆鲁棒性的代理。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable performance across diverse natural language processing tasks, yet their ability to memorize structured knowledge remains underexplored. In this paper, we investigate the extent to which general-purpose pre-trained LLMs retain and correctly reproduce concept identifier (ID)-label associations from publicly available ontologies. We conduct a systematic evaluation across multiple ontological resources, including the Gene Ontology, Uberon, Wikidata, and ICD-10, using LLMs such as Pythia-12B, Gemini-1.5-Flash, GPT-3.5, and GPT-4. Our findings reveal that only a small fraction of ontological concepts is accurately memorized, with GPT-4 demonstrating the highest performance. To understand why certain concepts are memorized more effectively than others, we analyze the relationship between memorization accuracy and concept popularity on the Web. Our results indicate a strong correlation between the frequency of a concept's occurrence online and the likelihood of accurately retrieving its ID from the label. This suggests that LLMs primarily acquire such knowledge through indirect textual exposure rather than directly from structured ontological resources. Furthermore, we introduce new metrics to quantify prediction invariance, demonstrating that the stability of model responses across variations in prompt language and temperature settings can serve as a proxy for estimating memorization robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑