arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.13863cs.CLcs.AI
收藏

CodeGraph:用代码增强 LLM 的图推理能力

CodeGraph: Enhancing Graph Reasoning of LLMs with Code

HKUST(香港科技大学)

Qiaolong Cai, Zhaowei Wang, Shizhe Diao, James Kwok, Yangqiu Song

AI总结:

本文提出 CodeGraph 方法,将图问题解编码为可执行代码,通过从示例学习、生成并执行程序来解决图推理任务,在 GraphQA 数据集上较现有方法显著提升了性能并增强了推理的可控性与可解释性。

摘要:

随着大型语言模型(LLMs)的日益普及,对基础图算法问题进行推理是评估其处理和推断复杂图推理任务能力的重要中间步骤。现有方法通常将图结构数据转换为文本描述,然后使用 LLMs 进行推理和计算。然而,LLMs 在基础图算法问题的算术部分经常产生计算错误,例如计算边的数量。此外,它们难以控制或理解推理过程的输出,引发了关于 LLMs 是否只是在猜测的担忧。在本文中,我们介绍了 CodeGraph,一种将图问题解决方案编码为代码的方法。该方法通过从示例中学习、生成程序,并通过程序解释器执行它们来解决新的图问题。使用少样本设置,我们以 GPT-3.5 Turbo、Llama3-70B Instruct、Mixtral-8x22B Instruct 和 Mixtral-8x7B Instruct 作为基础 LLM 评估了 CodeGraph。在 GraphQA 数据集中使用六种图编码方法对六项任务的实验结果表明,CodeGraph 能将 LLMs 内部的图推理任务性能提升 1.3% 至 58.6%,具体取决于任务。与现有方法相比,CodeGraph 在图任务的算术问题上表现出色,并为推理过程提供了一种更具可控性和可解释性的方法。

英文摘要:

With the increasing popularity of large language models (LLMs), reasoning on basic graph algorithm problems is an essential intermediate step in assessing their abilities to process and infer complex graph reasoning tasks. Existing methods usually convert graph-structured data to textual descriptions and then use LLMs for reasoning and computation. However, LLMs often produce computation errors on arithmetic parts in basic graph algorithm problems, such as counting number of edges. In addition, they struggle to control or understand the output of the reasoning process, raising concerns about whether LLMs are simply guessing. In this paper, we introduce CodeGraph, a method that encodes graph problem solutions as code. The methods solve new graph problems by learning from exemplars, generating programs, and executing them via a program interpreter. Using the few-shot setting, we evaluate CodeGraph with the base LLM being GPT-3.5 Turbo, Llama3-70B Instruct, Mixtral-8x22B Instruct, and Mixtral-8x7B Instruct. Experimental results on six tasks with six graph encoding methods in the GraphQA dataset demonstrate that CodeGraph can boost performance on graph reasoning tasks inside LLMs by 1.3% to 58.6%, depending on the task. Compared to the existing methods, CodeGraph demonstrates strong performance on arithmetic problems in graph tasks and offers a more controllable and interpretable approach to the reasoning process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑