CSEPrompts:一个计算机科学入门提示词基准
CSEPrompts: A Benchmark of Introductory Computer Science Prompts
AI总结:
本文构建 CSEPrompts 基准,汇集计算机科学入门课程中的编程题与选择题,并通过实验评估多个 LLM 生成 Python 代码及回答基础问题的能力。
摘要:
人工智能、机器学习和自然语言处理(NLP)的最新进展推动了新一代大型语言模型(LLM)的发展,这些模型基于海量数据训练,通常拥有数万亿个参数。商业应用(例如 ChatGPT)已使这项技术面向公众开放,从而使利用 LLM 生成学术和专业用途的高质量文本成为可能。中小学和大学已经意识到学生越来越多地使用人工智能生成内容,并一直在研究这项新技术的影响及其潜在滥用。计算机科学(CS)及相关领域的教育项目受到的影响尤为突出,因为 LLM 还能够用多种编程语言生成程序代码。为帮助理解公开可用的 LLM 对 CS 教育的潜在影响,我们提出了 CSEPrompts,这是一个包含数百个编程练习提示词和多项选择题的框架,这些题目取自 CS 与编程入门课程。我们还给出了在 CSEPrompts 上的实验结果,以评估若干 LLM 在生成 Python 代码以及回答基础计算机科学和编程问题方面的性能。
英文摘要:
Recent advances in AI, machine learning, and NLP have led to the development of a new generation of Large Language Models (LLMs) that are trained on massive amounts of data and often have trillions of parameters. Commercial applications (e.g., ChatGPT) have made this technology available to the general public, thus making it possible to use LLMs to produce high-quality texts for academic and professional purposes. Schools and universities are aware of the increasing use of AI-generated content by students and they have been researching the impact of this new technology and its potential misuse. Educational programs in Computer Science (CS) and related fields are particularly affected because LLMs are also capable of generating programming code in various programming languages. To help understand the potential impact of publicly available LLMs in CS education, we introduce CSEPrompts, a framework with hundreds of programming exercise prompts and multiple-choice questions retrieved from introductory CS and programming courses. We also provide experimental results on CSEPrompts to evaluate the performance of several LLMs with respect to generating Python code and answering basic computer science and programming questions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。