ComplexCodeEval：一个用于在更复杂代码上评估大代码模型的基准
ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
AI总结：
针对现有代码模型评估基准场景单一的问题，提出覆盖多类开发任务、含真实仓库标注样本的ComplexCodeEval基准，实验发现上下文可提升性能、数据泄露会致评估高估。
摘要：
近年来，大语言模型（LLMs）在代码相关任务中的应用受到广泛关注。然而，现有评估基准往往聚焦于代码生成或补全等有限场景，无法反映开发者在真实场景中面临的多样化挑战。为解决这一问题，我们提出ComplexCodeEval（复杂代码评估基准），该基准旨在评估大代码模型（LCMs）在多种开发任务中的表现，涵盖代码生成、补全、API推荐及测试用例生成。它包含来自高星GitHub仓库的3897个Java样本和7184个Python样本，每个样本都标注了函数签名、文档字符串及API参考，以模拟真实开发环境。我们在10个LCMs上开展的实验表明，上下文能提升性能，且数据泄露会导致评估结果高估，凸显了开展更准确评估的必要性。
英文摘要：
In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
