arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.21071cs.SE
收藏

代码任务基准的自动生成与可靠的 LLM 评判

Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks

Eitan Farchi, Shmulik Froimovich, Rami Katan, Orna Raz

AI总结:

本文提出自动生成多语言代码任务基准的方法,并通过生成图 G、LLM 智能体链与图中环形成的预期,开发和验证可靠的 LaaJ 代码产物有用性评判。

摘要:

LLM 可用于多种代码相关任务,例如将一种编程语言翻译成另一种编程语言、实现自然语言需求以及代码摘要。由最先进的 LLM 技术生成的产物,预期应在如下意义上是有用的:用户只需进行少量简单修改,便能够使用 LLM 生成的产物。量化这一模糊概念具有挑战性,因此很难确定代码相关 LLM 解决方案的质量。我们将使用 LLM 判断来评估 LLM 解决方案称为“LLM as a Judge”,简称 LaaJ。在本工作中,我们引入了一种生成和评估 LaaJ 实现的方法,该方法利用自动生成的基准。该基准的目的具有两重性:既用于开发和验证 LaaJ,也用于借助 LaaJ 验证和测试 LLM 的代码相关解决方案。为此,我们开发了一个自动化基准生成引擎,该引擎为多种代码相关任务生成多种编程语言的代码,并作为 LaaJ 评估的输入。我们利用潜在代码相关生成结果的图表示 G。图的顶点是生成的产物,边表示可能的生成过程,例如根据自然语言需求生成 Java 程序。利用一连串 LLM 智能体和 G,我们生成代码相关产物。借助 G 中的环,我们对生成产物形成预期。利用这些形式化的预期,可以开发并测试可靠的 LLM 判断,以评估解决方案所生成产物的有用性。我们的方法能够创建高质量的代码任务解决方案。

英文摘要:

LLMs can be used in a variety of code related tasks such as translating from one programming language to another, implementing natural language requirements and code summarization. Artifacts generated by state of the art LLM technology are expected to be useful in the sense that a user will be able to use the LLM generated artifact after a small number of easy modifications. Quantifying this vague notion is challenging and it is thus hard to determine the quality of code related LLM solutions. We refer to evaluation of LLM solutions using LLM judgment as "LLM as a Judge", or LaaJ for short. In this work we introduce a methodology to generate and evaluate LaaJ implementations, utilizing an automatically generated benchmark. The purpose of the benchmark is two fold, namely, it is used both to develop and validate the LaaJs and to validate and test the LLM code related solution using the LaaJs. To that end, we developed an automated benchmark generation engine, which generates code in multiple programming languages for multiple code related tasks and which serves as the input for LaaJ evaluation. We utilize a graph representation, G, of the potential code related generations. The graph vertices are generated artifacts and edges represent possible generations, e.g., the generation of a Java program from its natural language requirements. Utilizing a chain of LLM agents and G we generate code related artifacts. Using cycles in G we formulate expectations on the generated artifacts. Taking advantage of these formulated expectations enables the development and testing of reliable LLM judgement for usefulness of the artifacts generated by the solution. Our approach enables the creation of high quality code task solutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑