在图上评估大型语言模型：性能洞察与比较分析
Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis
AI总结：
本研究评估了四种大型语言模型在图数据分析上的能力，采用理解力、正确性、保真度和纠正能力四项指标，发现LLMs能理解图数据但结构推理受限，GPT模型正确性更优但多答案保真度存疑，且GPT-4具备纠正先前版本回复的能力。
摘要：
大型语言模型（LLMs）在学术界和工业界引起了广泛关注。然而，LLMs在图数据上的应用仍未被充分探索。在本研究中，我们评估了四种LLMs在解决图数据的若干分析问题上的能力。我们采用了四种不同的评估指标：理解力、正确性、保真度和纠正能力。我们的结果显示：1）LLMs能够有效理解自然语言描述的图数据，并基于图拓扑进行推理。2）GPT模型能够生成逻辑连贯的结果，在正确性上优于其他替代方案。3）所有受检的LLMs在结构推理方面都面临挑战，零样本思维链和少样本提示等技术显示出减弱的效能。4）GPT模型在多答案任务中经常产生错误答案，引发了对保真度的担忧。5）GPT模型对其输出表现出较高的信心，这可能阻碍其纠正能力。值得注意的是，GPT-4已展现出纠正GPT-3.5-turbo及其自身先前迭代版本回复的能力。代码可在以下地址获取：https://github.com/Ayame1006/LLMtoGraph。
英文摘要：
Large Language Models (LLMs) have garnered considerable interest within both academic and industrial. Yet, the application of LLMs to graph data remains under-explored. In this study, we evaluate the capabilities of four LLMs in addressing several analytical problems with graph data. We employ four distinct evaluation metrics: Comprehension, Correctness, Fidelity, and Rectification. Our results show that: 1) LLMs effectively comprehend graph data in natural language and reason with graph topology. 2) GPT models can generate logical and coherent results, outperforming alternatives in correctness. 3) All examined LLMs face challenges in structural reasoning, with techniques like zero-shot chain-of-thought and few-shot prompting showing diminished efficacy. 4) GPT models often produce erroneous answers in multi-answer tasks, raising concerns in fidelity. 5) GPT models exhibit elevated confidence in their outputs, potentially hindering their rectification capacities. Notably, GPT-4 has demonstrated the capacity to rectify responses from GPT-3.5-turbo and its own previous iterations. The code is available at: https://github.com/Ayame1006/LLMtoGraph.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
