arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.00795cs.CLcs.AI
收藏

用大语言模型执行自然语言描述的算法:一项研究

Executing Natural Language-Described Algorithms with Large Language Models: An Investigation

University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)

Xin Zheng, Qiming Zhu, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun

AI总结:

本研究通过算法测试集评估大语言模型执行自然语言描述算法的能力,发现GPT-4等模型在无重数值计算时可有效执行,贡献于代码执行能力评估。

摘要:

用自然语言描述计算机程序并执行,长期以来一直是计算机科学追求的目标。随着大语言模型(LLMs)展现出增强的自然语言理解能力,实现这一目标的道路已被照亮。本文旨在考察当前大语言模型理解和执行以自然语言描述的算法的能力。我们建立了一个算法测试集,来源于著名教科书《算法导论》(Introduction to Algorithm),其中包含许多具有代表性的广泛使用的算法。为系统评估LLMs的代码执行能力,我们选取了30个算法,共生成300个随机采样实例,并评估了主流LLMs能否理解并执行这些算法。我们的发现表明,LLMs,尤其是GPT-4,能够有效执行自然语言描述的程序,只要不涉及繁重的数值计算。我们相信,我们的发现有助于评估LLMs的代码执行能力,并将鼓励进一步研究和应用LLMs的计算能力。

英文摘要:

Executing computer programs described in natural language has long been a pursuit of computer science. With the advent of enhanced natural language understanding capabilities exhibited by large language models (LLMs), the path toward this goal has been illuminated. In this paper, we seek to examine the capacity of present-day LLMs to comprehend and execute algorithms outlined in natural language. We established an algorithm test set sourced from Introduction to Algorithm, a well-known textbook that contains many representative widely-used algorithms. To systematically assess LLMs' code execution abilities, we selected 30 algorithms, generated 300 random-sampled instances in total, and evaluated whether popular LLMs can understand and execute these algorithms. Our findings reveal that LLMs, notably GPT-4, can effectively execute programs described in natural language, as long as no heavy numeric computation is involved. We believe our findings contribute to evaluating LLMs' code execution abilities and would encourage further investigation and application for the computation power of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑