arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.01784cs.AIcs.SE
收藏

通过 CodeWars 的 Katas 评估 GPT 的编程能力

Evaluating GPT's Programming Capability through CodeWars' Katas

Griffith University(格里菲斯大学)

Zizhuo Zhang, Lian Wen, Shaoyang Zhang, David Chen, Yanfei Jiang

AI总结:

本文用 Codewars 不同难度 Katas 评估 GPT-3.5 与 GPT-4,发现其能力边界在 3kyu,并提出结合难度与解题时间的编程题复杂度度量。

摘要:

在人工智能(AI)这一蓬勃发展的领域中,理解面向编程的模型的能力与局限至关重要。本文针对生成式预训练 Transformer(GPT)模型,特别是 GPT-3.5 和 GPT-4,使用取自 Codewars 的不同难度级别的编程题目,对其编程熟练度进行了一项新颖评估。实验揭示,在 3kyu 级别存在一个明显边界;超过该级别后,这些 GPT 模型便难以给出解决方案。基于这些发现,本文提出了一种衡量编程题目复杂度的方法,该方法同时纳入题目难度和求解所需时间。研究强调,AI 模型需要具备验证能力和创造性思维能力,以更好地模拟人类的问题解决技巧。未来工作旨在完善所提出的复杂度度量,用上述建议能力增强 AI 模型,并开发一种衡量编程题目难度的客观方法。本研究结果为提升 AI 编程能力、推进 AI 问题解决能力的前沿提供了宝贵洞见。

英文摘要:

In the burgeoning field of artificial intelligence (AI), understanding the capabilities and limitations of programming-oriented models is crucial. This paper presents a novel evaluation of the programming proficiency of Generative Pretrained Transformer (GPT) models, specifically GPT-3.5 and GPT-4, against coding problems of varying difficulty levels drawn from Codewars. The experiments reveal a distinct boundary at the 3kyu level, beyond which these GPT models struggle to provide solutions. These findings led to the proposal of a measure for coding problem complexity that incorporates both problem difficulty and the time required for solution. The research emphasizes the need for validation and creative thinking capabilities in AI models to better emulate human problem-solving techniques. Future work aims to refine this proposed complexity measure, enhance AI models with these suggested capabilities, and develop an objective measure for programming problem difficulty. The results of this research offer invaluable insights for improving AI programming capabilities and advancing the frontier of AI problem-solving abilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑