语言模型能用自然语言解决图问题吗？
Can Language Models Solve Graph Problems in Natural Language?
AI总结：
提出NLGraph基准评估LLMs的图推理能力，并设计Build-a-Graph和Algorithmic提示方法，在多任务上提升模型性能3.07%至16.85%。
摘要：
大型语言模型（LLMs）越来越多地被应用于各种具有隐式图结构的任务中，例如机器人规划、多跳问答或知识探测、结构化常识推理等。尽管LLMs在这些具有结构含义的任务上推进了最先进水平，但LLMs能否显式处理图和结构的文本描述，将它们映射到基础的概念空间，并执行结构化操作，仍有待探索。为此，我们提出了NLGraph（Natural Language Graph，自然语言图），这是一个用自然语言设计的基于图的问题求解的综合基准。NLGraph包含29,370个问题，涵盖八种复杂程度各异的图推理任务，从连通性和最短路径等简单任务，一直到最大流和模拟图神经网络等复杂问题。我们在NLGraph基准上评估了采用各种提示方法的LLMs（GPT-3/4），发现：1）语言模型确实展现出了初步的图推理能力；2）高级提示和上下文学习在更复杂的图问题上的收益会递减；3）LLMs在面对图和问题设置中的虚假相关性时，也（不）出人意料地脆弱。随后，我们提出了Build-a-Graph Prompting和Algorithmic Prompting这两种基于指令的方法，以增强LLMs解决自然语言图问题的能力。Build-a-Graph和Algorithmic提示在多个任务和设置下将LLMs在NLGraph上的性能提升了3.07%到16.85%，而如何在我们当前的设置下用语言模型解决最复杂的图推理任务，仍然是一个开放的研究问题。NLGraph基准和评估代码可在 https://github.com/Arthur-Heng/NLGraph 获取。
英文摘要：
Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures, such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense reasoning, and more. While LLMs have advanced the state-of-the-art on these tasks with structure implications, whether LLMs could explicitly process textual descriptions of graphs and structures, map them to grounded conceptual spaces, and perform structured operations remains underexplored. To this end, we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language. NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks. We evaluate LLMs (GPT-3/4) with various prompting approaches on the NLGraph benchmark and find that 1) language models do demonstrate preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning diminishes on more complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the face of spurious correlations in graph and problem settings. We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems. Build-a-Graph and Algorithmic prompting improve the performance of LLMs on NLGraph by 3.07% to 16.85% across multiple tasks and settings, while how to solve the most complicated graph reasoning tasks in our setup with language models remains an open research question. The NLGraph benchmark and evaluation code are available at https://github.com/Arthur-Heng/NLGraph.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
