arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23047cs.SE
收藏

CL4SE:软件工程中基于上下文学习的基准测试

CL4SE: Benchmarking Context Learning on Software Engineering

Nanjing University of Science and Technology(南京理工大学);Nanjing University(南京大学)

Haichuan Hu, Quanjun Zhang, Ye Shang, Guoqing Xie, Chunrong Fang, Zhenyu Chen, Liang Xiao

AI总结:

本文提出CL4SE基准测试,系统分类四种软件工程特定上下文类型,通过高质量数据集和五种主流LLM评估,展示上下文学习在代码生成、审查等任务中的性能提升。

摘要:

上下文工程已成为解锁大型语言模型(LLMs)在软件工程(SE)任务中潜力的关键范式,能够在测试时实现性能提升而无需模型微调。尽管已取得成功,现有研究缺乏对SE特定上下文类型的系统分类以及专门用于量化不同上下文在核心SE工作流中异质影响的基准测试。为此,我们提出了CL4SE(软件工程中的上下文学习),一个全面的基准测试,包含四种面向软件工程的上下文类型(可解释示例、项目特定上下文、过程决策上下文以及正负上下文)的细粒度分类,每种上下文映射到一个代表性任务(代码生成、代码摘要、代码审查和补丁正确性评估)。我们构建了一个包含超过13000个样本来自超过30个开源项目的高质量数据集,并在九个指标上评估了五种主流LLM。广泛实验表明,上下文学习在所有任务中平均性能提升了24.7%。具体而言,过程上下文使代码审查性能提升高达33%(Qwen3-Max),混合正负上下文使补丁评估提升30%(DeepSeek-V3),项目特定上下文使代码摘要BLEU提升14.78%(GPT-Oss-120B),可解释示例使代码生成PASS@1提升5.72%(DeepSeek-V3)。CL4SE建立了软件工程上下文学习的第一个标准化评估框架,提供了任务特定上下文设计的可操作实证见解,并发布了一个大规模数据集以促进该领域可重复研究。

英文摘要:

Context engineering has emerged as a pivotal paradigm for unlocking the potential of Large Language Models (LLMs) in Software Engineering (SE) tasks, enabling performance gains at test time without model fine-tuning. Despite its success, existing research lacks a systematic taxonomy of SE-specific context types and a dedicated benchmark to quantify the heterogeneous effects of different contexts across core SE workflows. To address this gap, we propose CL4SE (Context Learning for Software Engineering), a comprehensive benchmark featuring a fine-grained taxonomy of four SE-oriented context types (interpretable examples, project-specific context, procedural decision-making context, and positive & negative context), each mapped to a representative task (code generation, code summarization, code review, and patch correctness assessment). We construct high-quality datasets comprising over 13,000 samples from more than 30 open-source projects and evaluate five mainstream LLMs across nine metrics. Extensive experiments demonstrate that context learning yields an average performance improvement of 24.7% across all tasks. Specifically, procedural context boosts code review performance by up to 33% (Qwen3-Max), mixed positive-negative context improves patch assessment by 30% (DeepSeek-V3), project-specific context increases code summarization BLEU by 14.78% (GPT-Oss-120B), and interpretable examples enhance code generation PASS@1 by 5.72% (DeepSeek-V3). CL4SE establishes the first standardized evaluation framework for SE context learning, provides actionable empirical insights into task-specific context design, and releases a large-scale dataset to facilitate reproducible research in this domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑