arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.02235cs.SE
收藏

基于代理的软件工件评估

Agent-Based Software Artifact Evaluation

Huazhong University of Science and Technology(华中科技大学);Tsinghua University(清华大学)

Zhaonan Wu, Yanjie Zhao, Zhenpeng Chen, Zheng Wang, Haoyu Wang

AI总结:

研究针对软件工件评估中人工评估难及现有政策缺乏验证标准的问题,构建ArtifactGuide评分标准,设计ArtifactCopilot代理,通过实验评估,该框架提升了评估性能,提高了评审信心,还为设计高质量工件提供了实践指导。

摘要:

工件评估(AE)已成为将软件工程论文与其支持工件相联系的标准机制,但提交量增加使纯人工AE难以持续。尽管基于大语言模型的代理取得进展使AE自动化更可行,但当前描述性徽章政策缺乏可执行验证标准。为此构建了基于ACM政策、专家校准和工件验证的结构化评分标准ArtifactGuide,设计了按固定序列收集审查证据并据此做出最终徽章决定的代理ArtifactCopilot。通过60个真实工件评估,结果显示ArtifactGuide提高了编码代理的AE性能,ArtifactCopilot徽章级别一致性最高,用户研究表明其报告提升了评审信心等,还为设计高质量工件提供了实用指导。

英文摘要:

Artifact Evaluation (AE) has become a standard mechanism for linking software engineering papers to their supporting artifacts, but growing submission volume makes purely manual AE increasingly difficult to sustain. Although recent advances in LLM-based agents make AE automation increasingly plausible, current descriptive badge policies define badge semantics but provide no executable verification criteria, leaving both human reviewers and agents without a detailed basis for judgment. To address this gap, we construct ArtifactGuide, a structured scoring rubric grounded in ACM policy, expert-informed calibration, and artifact-based validation, and we design ArtifactCopilot, an agent collecting review evidence following a fixed sequence under ArtifactGuide and deriving the final badge decision from accumulated evidence. We evaluate our framework on 60 real artifacts from recent software engineering conferences using human-adjudicated badges as reference. The results show that ArtifactGuide improves the AE performance of coding agents over official ACM badge-policy prompts, increasing three-run mean exact badge agreement by 10.55 to 28.34 percentage points. Across all evaluated systems and prompting protocols, ArtifactCopilot achieves the highest badge-level agreement at 70.56% and is the only system that completed all repeated runs successfully while producing a review report in every run. A controlled user study with 8 experienced researchers suggests that ArtifactCopilot reports improve reviewer confidence, help reviewers locate evidence, and understand evaluation scope more clearly. Further analysis translates insights from automated AE into practical guidance for designing higher-quality artifacts with clearer review routes, more explicit claim-to-output links, and more concrete reuse paths.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑