SWE-Bench-CL：面向编码智能体的持续学习基准
SWE-Bench-CL: Continual Learning for Coding Agents
AI总结：
本文提出 SWE-Bench-CL 持续学习基准，基于 SWE-Bench Verified 按时间组织 GitHub 议题序列，并配套语义记忆评估框架与多维度持续学习指标，用于评估编码智能体的经验积累、知识迁移与抗遗忘能力。
摘要：
大型语言模型（LLMs）在静态代码生成基准上取得了令人瞩目的成果，但现实世界的软件开发呈现为不断演进的议题、修复和功能请求的连续流。我们引入了 SWE-Bench-CL，这是一个新颖的持续学习基准，构建于 OpenAI 与 Princeton-NLP 于 2024 年发布的人工验证的 SWE-Bench Verified 数据集之上。通过将 GitHub 议题组织成反映自然仓库演进的按时间排序的序列，SWE-Bench-CL 能够直接评估智能体积累经验、跨任务迁移知识以及抵抗灾难性遗忘的能力。我们为该数据集补充了：（i）对任务间结构相似性与上下文敏感性的初步分析；（ii）一个基于 LangGraph 的交互式评估框架，并增配了一个由 FAISS 支持的语义记忆模块；以及（iii）一套专门的持续学习指标——包括平均准确率、遗忘、前向/后向迁移、工具使用效率，以及一个广义的复合持续学习分数（Composite Continual Learning Score）和 CL-F-beta 分数——以刻画稳定性-可塑性权衡。我们概述了一个严格的实验协议，在多个不同的 Python 仓库上比较启用记忆与禁用记忆的智能体。所有代码和数据均公开于 https://github.com/thomasjoshi/agents-never-forget，为社区提供了一个可复现的平台，用于在软件工程中开发更具适应性和鲁棒性的 AI 智能体。
英文摘要：
Large Language Models (LLMs) have achieved impressive results on static code-generation benchmarks, but real-world software development unfolds as a continuous stream of evolving issues, fixes, and feature requests. We introduce SWE-Bench-CL, a novel continual learning benchmark built on the human-verified SWE-Bench Verified dataset introduced by OpenAI and Princeton-NLP in 2024. By organizing GitHub issues into chronologically ordered sequences that reflect natural repository evolution, SWE-Bench-CL enables direct evaluation of an agent's ability to accumulate experience, transfer knowledge across tasks, and resist catastrophic forgetting. We complement the dataset with (i) a preliminary analysis of inter-task structural similarity and contextual sensitivity, (ii) an interactive LangGraph-based evaluation framework augmented with a FAISS-backed semantic memory module, and (iii) a suite of specialized continual learning metrics -- including average accuracy, forgetting, forward/backward transfer, tool-use efficiency, and a generalized Composite Continual Learning Score and CL-F-beta score -- to capture the stability-plasticity trade-off. We outline a rigorous experimental protocol comparing memory-enabled and memory-disabled agents across diverse Python repositories. All code and data are publicly available at https://github.com/thomasjoshi/agents-never-forget, providing the community with a reproducible platform for developing more adaptive and robust AI agents in software engineering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
