arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.16528cs.CL
收藏

评估大型语言模型对基本方位的推理能力

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions

School of Computing, University of Leeds(利兹大学计算学院);The Alan Turing Institute(阿兰·图灵研究所);Microsoft Research(微软研究院)

Anthony G Cohn, Robert E Blackwell

AI总结:

本文构建知识回忆与模板场景两个数据集,评估代表性 LLM 的基本方位推理能力,发现其在复杂场景中即使温度为零也无法可靠判断。

摘要:

我们研究了一组具有代表性的大型语言模型(LLM)对基本方位(CD)进行推理的能力。为此,我们创建了两个数据集:第一个与 ChatGPT 共同构建,主要侧重于回忆有关 CD 的世界知识;第二个由一组模板生成,全面测试 LLM 在特定场景下确定正确 CD 的能力。这些模板允许多种程度的变化,例如所涉智能体的移动方式,以及场景设定为第一人称、第二人称还是第三人称。我们的实验表明,即使在温度设置为零的情况下,尽管 LLM 能够在较简单的数据集上表现良好,但在第二个更复杂的数据集中,没有任何 LLM 能够可靠地确定正确的 CD,即便温度设置为零也是如此。

英文摘要:

We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first , second or third person. Even with a temperature setting of zero, Our experiments show that although LLMs are able to perform well in the simpler dataset, in the second more complex dataset no LLM is able to reliably determine the correct CD, even with a temperature setting of zero.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑