ConstraintChecker:一个用于大语言模型在常识知识库上进行推理的插件
ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases
AI总结:
针对大语言模型难以从示例中掌握常识知识库显式关系约束的问题,本文提出 ConstraintChecker 插件,通过规则生成约束并用零样本模块检查,再与提示输出聚合,在多个基准上一致提升性能。
摘要:
在常识知识库(Commonsense Knowledge Bases,CSKB)上进行推理,即 CSKB 推理,已被探索为一种基于原始 CSKB 中的参考知识和外部先验知识获取新常识知识的方法。尽管大语言模型(LLM)和提示工程技术在各类推理任务中取得了进展,它们仍难以处理 CSKB 推理。问题之一在于,由于缺乏符号推理能力(Bengio et al., 2021),它们很难仅凭上下文中的示例获得 CSKB 中的显式关系约束。为此,我们提出了 ConstraintChecker,这是一个构建于提示技术之上的插件,用于提供并检查显式约束。在考虑一个新的知识实例时,ConstraintChecker 使用基于规则的模块生成约束列表,然后使用零样本学习模块检查该知识实例是否满足所有约束。随后,将获得的约束检查结果与主要提示技术的输出聚合,以生成最终输出。在 CSKB 推理基准上的实验结果表明,我们的方法能够为所有提示方法带来一致提升,证明了其有效性。代码和数据可在 https://github.com/HKUST-KnowComp/ConstraintChecker 获取。
英文摘要:
Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge based on reference knowledge in the original CSKBs and external prior knowledge. Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning. One of the problems is that it is hard for them to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to a lack of symbolic reasoning capabilities (Bengio et al., 2021). To this end, we proposed **ConstraintChecker**, a plugin over prompting techniques to provide and check explicit constraints. When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints. The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output. Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all prompting methods. Codes and data are available at \url{https://github.com/HKUST-KnowComp/ConstraintChecker}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。