CodeEval: 一种面向代码训练大语言模型的教育性评估方法
CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
AI总结:
CodeEval通过多维基准数据集和开源执行框架,针对代码训练大语言模型的评估与改进提供教育性方法。
摘要:
大型语言模型(LLMs)主要基于常识推理、语言理解和逻辑推理能力进行评估。虽然在数学或编码等专业领域训练的模型在逻辑推理方面取得了显著进展,但对其代码生成能力的评估仍存在明显差距。现有基准数据集在定位特定优势和劣势方面表现不足,阻碍了模型推理能力在合成代码方面的针对性提升。为此,本文提出了一种创新的教育性基准评估方法,模拟学术编程课程中遇到的评估过程。我们引入了CodeEval,一个多维基准数据集,旨在严格评估LLMs在Python编程的24个不同方面的表现。该数据集涵盖初级、中级和高级三个熟练程度级别,并包含基于类和函数的问题类型,具有详细的问题规范和全面的测试套件。为了促进广泛应用,我们还开发了RunCodeEval,一个开源执行框架,为研究人员提供了可以直接使用的评估流程。RunCodeEval处理测试执行、上下文设置和指标生成,使研究人员能够快速获得模型在复杂程度、问题类型和编程类别上的详细洞察。这种结合使能够实现有针对性的评估,并指导LLMs编程能力的改进。
英文摘要:
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models' reasoning abilities to synthesize code. To bridge this gap, our paper introduces an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. We introduce CodeEval, a multi-dimensional benchmark dataset designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming. The dataset covers three proficiency levels - beginner, intermediate, and advanced - and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites. To facilitate widespread adoption, we also developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for CodeEval. RunCodeEval handles test execution, context setup, and metrics generation, enabling researchers to quickly obtain detailed insights into model strengths and weaknesses across complexity levels, problem types, and programming categories. This combination enables targeted evaluation and guides improvements in LLMs' programming proficiencies.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。