面向仓库的长 horizon 对话上下文管理可扩展基准
A Scalable Benchmark for Repository-Oriented Long-Horizon Conversational Context Management
AI总结：
本文提出LoCoEval，首个面向仓库开发场景的长 horizon 对话上下文管理基准，评估了多种方法并提出改进方案，提升了代码助手在复杂对话中的表现。
摘要：
近年来，大语言模型（LLMs）发展迅速，显著增强了其代码理解和生成能力，催生了强大的代码助手。然而，在实际的仓库开发中，过长的对话上下文可能使模型不堪重负，导致关键信息丢失和性能下降，从而限制了代码助手的实用性。现有的上下文管理方法主要针对通用对话，而面向仓库的解决方案仍鲜有探索，这主要由于缺乏可靠的评估基准。为填补这一空白，我们提出了LoCoEval，首个面向仓库开发场景的长 horizon 对话上下文管理基准。遵循三个关键原则，LoCoEval通过LLM驱动的流程生成真实且多样的仓库对话，捕捉关键交互模式，如迭代需求、噪声输入和回顾性问题。我们使用3种先进的基础LLMs评估了7个基线，包括4个代表性上下文管理方法。结果揭示了独立LLMs和现有方法在仓库对话场景中的显著挑战，尤其是内存系统。为解决这些限制，我们进一步提出一种改进方法，将对话和仓库信息整合到统一的记忆中，该方法在所有基线（*Oracle*除外）中表现更优，并展示了鲁棒性。此外，我们还研究了各种因素对方法性能的影响，为未来研究提供了可行的见解。
英文摘要：
In recent years, large language models (LLMs) have advanced rapidly, substantially enhancing their code understanding and generation capabilities and giving rise to powerful code assistants. However, in practical repository development, excessively long-horizon conversational context may overwhelm models, causing the loss of critical information and degraded performance, thereby limiting the utility of code assistants. Existing context management methods proposed to mitigate this context dilemma primarily target general-purpose conversations, while repository-oriented solutions remain largely unexplored, which is largely due to the lack of reliable evaluation benchmarks. To bridge this gap, we present LoCoEval, the first long-horizon conversational context management benchmark tailored to repository-oriented development scenarios. Adhering to three key principles, LoCoEval is constructed via an LLM-driven pipeline that generates realistic and diverse repository-oriented conversations, capturing key interaction patterns such as iterative requirements, noisy input, and retrospective questions. We evaluate 7 baselines, including 4 representative context management methods, using 3 advanced backbone LLMs on LoCoEval. The results reveal substantial challenges faced by standalone LLMs and existing approaches, especially memory systems, in repository-oriented conversational scenarios. To address these limitations, we further propose an improved method integrating conversational and repository information into a unified memory, which outperforms all baselines (*Oracle* excluded) and demonstrates robustness. Additionally, we investigated the impact of various factors on method performance, providing actionable insights for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
