arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12902cs.LGcs.AIcs.PLcs.SE
收藏

JavaBench：用于评估大型语言模型的面向对象代码生成基准

JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models

The Hong Kong University of Science and Technology（香港科技大学）；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）；Nanjing University（南京大学）

Jialun Cao, Zhiyong Chen, Jiarong Wu, Shing-chi Cheung, Chang Xu

备注：
Accepted by ASE 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出项目级Java基准JavaBench，包含4个项目、106个类和389个方法，评估LLM面向对象代码生成能力，发现LLM在项目级Java编程上远落后于本科生。

摘要：

诸如 HumanEval 之类的代码生成基准被广泛采用以评估 LLMs 的能力。然而，在整合了最新的 24 个基准后，我们注意到三个显著的不平衡。首先，编程语言不平衡。95.8% 的基准涉及 Python，而仅有 5 个基准涉及 Java。其次，代码粒度不平衡。函数/语句级基准占 83.3% 以上。仅有极少数扩展到类/项目级别，且均局限于 Python。第三，缺乏高级特性。现有基准主要评估基本编码技能，而忽略了高级的面向对象编程（OOP）特性（即封装、继承和多态）。为了填补这些空白，我们提出了 JavaBench，一个运用 OOP 特性的项目级 Java 基准。它包含 4 个 Java 项目，在 106 个 Java 类中有 389 个方法。测试覆盖率高达 92%，并且 JavaBench 经过了 282 名本科生的验证，达到了 90.93/100 的平均分（即通过测试套件的通过率），确保了文档、代码骨架和测试的质量。为了更好地评估 LLM 针对 JavaBench 的能力，我们引入了一种系统的评估设计，涵盖在两种粒度下的三种上下文设置和五种合成策略，并使用三种层次化指标。我们广泛的实验得出了几个有趣的发现。首先，我们注意到在项目级 Java 编程方面，LLMs 远落后于本科生（没有任何被研究的 LLM 能正确完成任何项目，在更宽松的评估中最多达到 41.17% 的 Pass@5）。其次，使用方法签名作为提示上下文可能会在项目级代码生成中取得理想的平衡。JavaBench 已在 https://github.com/java-bench/JavaBench 公开。

英文摘要：

Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs' capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant imbalances. First, imbalanced programming language. 95.8% of benchmarks involve Python, while only 5 benchmarks involve Java. Second, imbalanced code granularity. Function-/statement-level benchmarks account for over 83.3% of benchmarks. Only a mere handful extends to class-/project-levels, and all are limited to Python. Third, lacking advanced features. Existing benchmarks primarily assess basic coding skills, while overlooking advanced Object-Oriented Programming (OOP) features (i.e., encapsulation, inheritance, and polymorphism). To fill these gaps, we propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. The test coverage is up to 92%, and JavaBench is attested by 282 undergraduate students, reaching a 90.93/100 average score (i.e., pass rate against the test suite), ensuring the quality of documentation, code skeleton, and tests. To better evaluate LLM's capability against JavaBench, we introduce a systematic evaluation design covering three context settings and five synthesis strategies at two granularities using three hierarchical metrics. Our extensive experiment yields several interesting findings. First, we noticed that regarding project-level Java programming, LLMs are far behind undergraduate students (no project can be correctly completed by any studied LLMs, and at most 41.17% Pass@5 in a more relaxed evaluation). Second, using method signature as prompt context may strike an ideal balance for project-level code generation. JavaBench is publicly available at https://github.com/java-bench/JavaBench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑