arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.20163cs.CLcs.AI
收藏

使用 LLMs 进行概念推理:不一致性普遍存在

Reasoning about concepts with LLMs: Inconsistencies abound

IBM Research(IBM研究院)

Rosario Uceda-Sosa, Karthikeyan Natesan Ramamurthy, Maria Chang, Moninder Singh

AI总结:

针对LLMs在概念推理中存在的不一致性问题,提出基于知识图谱的本体检测方法和提示策略,显著提升了不同规模开源LLMs的概念一致性与性能。

摘要:

将知识总结和组织为抽象概念的能力是学习和推理的关键。许多工业应用依赖于概念的一致和系统使用,特别是在处理决策关键知识时。然而,我们证明,当被系统地提问时,大型语言模型(LLMs)往往在其知识中显示并表现出显著的不一致性。在计算上,给定领域概念化的基本方面可以表示为知识图谱(KG)或本体中的 Is-A 层次结构,以及一些能够进行直接推理的属性或公理。我们表明,即使是简单的本体也可用于揭示几个 LLMs 中的概念不一致性。我们还提出了领域专家可用于评估和改进各种规模 LLMs 中关键领域概念覆盖范围的策略。特别是,我们能够使用基于简单知识图谱(KG)的提示策略,显著提升具有公开可用权重的各种规模 LLMs 的性能。

英文摘要:

The ability to summarize and organize knowledge into abstract concepts is key to learning and reasoning. Many industrial applications rely on the consistent and systematic use of concepts, especially when dealing with decision-critical knowledge. However, we demonstrate that, when methodically questioned, large language models (LLMs) often display and demonstrate significant inconsistencies in their knowledge. Computationally, the basic aspects of the conceptualization of a given domain can be represented as Is-A hierarchies in a knowledge graph (KG) or ontology, together with a few properties or axioms that enable straightforward reasoning. We show that even simple ontologies can be used to reveal conceptual inconsistencies across several LLMs. We also propose strategies that domain experts can use to evaluate and improve the coverage of key domain concepts in LLMs of various sizes. In particular, we have been able to significantly enhance the performance of LLMs of various sizes with openly available weights using simple knowledge-graph (KG) based prompting strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑