arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.02432cs.CYcs.AIcs.CL
收藏

大型语言模型在计算机科学学位项目中的表现

Performance of Large Language Models in a Computer Science Degree Program

Niederrhein University of Applied Sciences(下莱茵应用科学大学)

Tim Krüger, Michael Gref

AI总结:

本文评估了大型语言模型在应用科学大学计算机科学本科项目中的表现,发现ChatGPT-3.5、BingAI和LLaMa在10个模块中平均得分分别为79.9%、68.4%和20%,但即使是GPT-4.0也因数学计算局限无法通过学位项目。

摘要:

诸如ChatGPT-3.5和GPT-4.0等大型语言模型无处不在,主导着当前的讨论。它们的变革性能力引发了我们在交互和使用(基于文本的)信息方式上的范式转变。每天都有利用这些模型能力的新可能性出现。本文介绍了一所应用科学大学本科计算机科学学位项目中不同大型语言模型表现的研究结果。我们的主要目标是通过将这些模型用作教育辅助工具来评估它们在课程中的有效性。通过用讲座材料、练习任务和过去的考试题目提示这些模型,我们旨在评估它们在不同计算机科学领域的熟练程度。我们展示了当前大型语言模型的强大表现,同时突出了此类学位项目背景下的局限性和约束条件。我们发现,ChatGPT-3.5在10个测试模块中平均得分占总分的79.9%,BingAI达到68.4%,而LLaMa(650亿参数版本)为20%。尽管这些结果令人信服,但即使是GPT-4.0也无法通过该学位项目——由于数学计算方面的局限性。

英文摘要:

Large language models such as ChatGPT-3.5 and GPT-4.0 are ubiquitous and dominate the current discourse. Their transformative capabilities have led to a paradigm shift in how we interact with and utilize (text-based) information. Each day, new possibilities to leverage the capabilities of these models emerge. This paper presents findings on the performance of different large language models in a university of applied sciences' undergraduate computer science degree program. Our primary objective is to assess the effectiveness of these models within the curriculum by employing them as educational aids. By prompting the models with lecture material, exercise tasks, and past exams, we aim to evaluate their proficiency across different computer science domains. We showcase the strong performance of current large language models while highlighting limitations and constraints within the context of such a degree program. We found that ChatGPT-3.5 averaged 79.9% of the total score in 10 tested modules, BingAI achieved 68.4%, and LLaMa, in the 65 billion parameter variant, 20%. Despite these convincing results, even GPT-4.0 would not pass the degree program - due to limitations in mathematical calculations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑