arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.25880cs.SE
收藏

从线程到轨迹:一个多LLM管道用于从GitHub问题讨论中提取社区知识

From Threads to Trajectories: A Multi-LLM Pipeline for Community Knowledge Extraction from GitHub Issue Discussions

The University of Texas at Dallas(德克萨斯大学达拉斯分校)

Nazia Shehnaz Joynab, Soneya Binta Hossain

AI总结:

本文提出SWE-MIMIC-Bench数据集,通过多LLM管道从GitHub讨论生成问题轨迹,用于提取复杂问题的结构化知识,提升软件工程社区的协作理解与LLM训练效果。

摘要:

在大规模开源软件项目中,解决复杂后期生产问题需要大量认知努力,因为开发者需遍历长且无结构的讨论线程。本文提出SWE-MIMIC-Bench,通过自动化多LLM管道从原始GitHub讨论生成问题轨迹。该管道利用多个闭源LLM执行细粒度任务:分析单个评论并考虑外部链接资源,将评论分析分类到特定领域(如根本原因、解决方案计划、实施进度),并合成标签感知的轨迹,捕捉讨论线程的结构化和连贯叙述。我们的管道使用五个闭源LLM配置用于不同目的:标签分类、内联代码块和外部链接摘要、评论分析、标签特定领域分类和轨迹合成。通过从复杂对话线程生成简洁可靠的轨迹,该系统可帮助软件工程社区的开发者和研究人员理解经验驱动的协作方法。此外,生成的轨迹可用于训练现代LLM代理,使其像专家开发者一样思考和行动。我们在SWE-Bench-Pro、SWE-Bench-Multilingual和SWE-Bench-Verified数据集中的800个真实GitHub问题上评估了该系统,成功提取734个高保真推理轨迹,成功率为91.7%。

英文摘要:

Resolution of complex post-production issues in large-scale open-source software (OSS) projects requires significant cognitive effort, as developers need to go through long, unstructured and fragmented issue discussion threads before that. In this paper, we present SWE-MIMIC-Bench, an issue trajectory dataset generated from raw GitHub discussions using an automated multi-LLM pipeline. Unlike simple summarization, this pipeline utilizes a group of closed-source LLMs to perform granular tasks: analyzing individual comments with awareness of externally-linked resources, classifying comment analyses into label-specific fields (e.g., root cause, solution plan, implementation progress), and synthesizing label-aware trajectories which capture a structured and coherent narrative of the entire discussion thread. Our pipeline uses five closed-source LLM configurations for distinct purposes: label classification, inline code block and external link summarization, comment analysis, label-specific field classification and trajectory synthesis. By generating concise and reliable trajectories from complex conversation threads, this system can assist developers and researchers of broader software engineering community to understand the experience-driven collaborative approach for issue diagnosis. Furthermore, the generated trajectories can be used to train modern LLM agents to think and act like an expert developer. We evaluated our system on 800 real-world GitHub issues drawn from the SWE-Bench-Pro, SWE-Bench-Multilingual and SWE-Bench-Verified dataset, achieving a 91.7% success rate in extracting 734 high-fidelity reasoning trajectories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑