ArkEval: 对ArkTS自动代码修复的基准测试与评估
ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
AI总结：
ArkEval为ArkTS自动代码修复提供首个全面基准，通过挖掘官方仓库问题并采用LLM测试机制，评估了四种先进模型的修复能力，揭示了LLM在该领域的现状与局限。
摘要：
大型语言模型已改变了代码生成，使软件开发的自动化达到了前所未有的水平。随着移动生态系统的演变，HarmonyOS已成为一个关键平台，需要强大的开发工具。HarmonyOS生态系统中的软件开发严重依赖于ArkTS，这是一种静态类型扩展的TypeScript。尽管其重要性日益增长，但生态系统缺乏强大的自动代码修复工具，主要是由于缺乏高质量的评估基准。为了解决这一缺口，我们提出了ArkEval，一个统一的框架，用于ArkTS自动修复工作流程的评估和基准建设。它提供了第一个全面的基准，专门针对ArkTS自动程序修复。我们通过挖掘包含超过400个独立ArkTS应用程序的大型官方华为仓库中的问题来构建该基准。通过严格的多阶段过滤过程，我们精选出502个可重复的问题。为了确保可测试性，我们采用了基于LLM的测试生成和投票机制，涉及Claude和其他模型。此外，我们标准化了问题陈述，以促进公平的评估。最后，我们使用检索增强的修复工作流程，在我们的基准上评估了四个最先进的大型语言模型（LLMs）。我们的结果突显了LLMs在修复ArkTS代码方面的当前能力和限制，为未来在低资源语言领域进行研究铺平了道路。
英文摘要：
Large language models have transformed code generation, enabling unprecedented automation in software development. As mobile ecosystems evolve, HarmonyOS has emerged as a critical platform requiring robust development tools. Software development for the HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript. Despite its growing importance, the ecosystem lacks robust tools for automated code repair, primarily due to the absence of a high-quality benchmark for evaluation. To address this gap, we present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction. It provides the first comprehensive benchmark specifically designed for ArkTS automated program repair. We constructed this benchmark by mining issues from a large-scale official Huawei repository containing over 400 independent ArkTS applications. Through a rigorous multi-stage filtering process, we curated 502 reproducible issues. To ensure testability, we employed a novel LLM-based test generation and voting mechanism involving Claude and other models. Furthermore, we standardized problem statements to facilitate fair evaluation. Finally, we evaluated four state-of-the-art Large Language Models (LLMs) on our benchmark using a retrieval-augmented repair workflow. Our results highlight the current capabilities and limitations of LLMs in repairing ArkTS code, paving the way for future research in this low-resource language domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
