arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.15976cs.AI
收藏

面向AI生成科学代码的PETSc代理评估框架

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

Argonne National Laboratory(阿贡国家实验室)

Hong Zhang, Barry Smith, Satish Balay, Le Chen, Murat Keceli, Lois Curfman McInnes, Junchao Zhang

AI总结:

本文提出petscagent-bench框架,通过代理评估代理的方式,评估AI生成的科学代码在正确性、性能、代码质量等方面的表现,揭示传统指标的不足。

摘要:

尽管大语言模型显著加速了科学代码生成,但全面评估生成代码仍面临挑战。传统基准仅依赖测试用例匹配,无法满足HPC库代码中求解器选择、API规范、内存管理及性能等关键因素。为此,我们引入petscagent-bench,基于代理评估代理的范式,部署工具增强的评估代理,编译、执行并测量由被测模型生成的代码,通过标准化协议(A2A和MCP)实现对任何编码代理的黑盒评估。我们在PETSc库的基准测试中展示了该框架,实证分析表明当前模型虽能生成可读性强的代码,但难以满足库特定规范,传统二元指标无法捕捉此问题。

英文摘要:

While LLMs have accelerated scientific code generation, comprehensively evaluating generated code remains challenging. Many benchmarks emphasize functional correctness or task completion, which is insufficient for code built on production HPC libraries, where solver selection, API conventions, memory management, parallel awareness, and performance also matter. We introduce PETSCAgent-Bench, a multidimensional benchmark and agent-based framework for assessing whether AI-generated scientific code uses a production HPC library as an expert would. A tool-augmented evaluator compiles, executes, and measures code and combines deterministic checks with LLM-based assessments in a 14-evaluator pipeline spanning five categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. A2A and MCP enable black-box evaluation of compatible coding agents. Across realistic PETSc problems, frontier models generate readable, well-structured code but struggle with correctness on challenging problems and with library-specific conventions even when code compiles and runs---limitations that conventional pass/fail evaluation does not capture.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑