面向IT自动化中自然语言转Bash的无参考自动代码验证与优化的LLM-as-a-Judge方法
LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation
AI总结：
针对IT自动化事件修复中代码验证需求，提出增强型LLM-as-a-Judge方法，结合双向功能匹配与逻辑表示实现无参考Bash代码验证，较基线准确率最高提升8%，构建的Reflection代码智能体可将代码优化准确率最高提升24%。
摘要：
为了在IT自动化的自动事件修复场景中自动评估、筛选最优模型并提升代码质量，验证生成的修复动作代码在语法、语义上的正确性以及能否按预期正确执行至关重要。现有三类方法：1）传统方法采用表层形式相似性指标（词元匹配、精确匹配等），存在诸多局限；2）基于执行的评估更聚焦代码功能，基于给定测试用例的通过/失败结果进行判断；3）LLM-as-a-Judge（大语言模型作为评判器）利用大语言模型开展自动评估，依据预定义指标判断给定问题的答案是否正确。本研究聚焦于增强LLM-as-a-Judge方法，通过双向功能匹配与逻辑表示，实现Bash代码生成的无参考自动验证与优化，以筛选出适用于IT自动化自动事件修复的最优模型。我们以基于执行的评估结果作为真值，来评测所提出的LLM-as-a-Judge指标。结果显示，该方法准确率高，与基于执行的评估结果一致性好，较基线最高提升8%。最后，我们构建了Reflection代码智能体，利用该评估指标的评判结果与反馈开展自动代码优化，实现了显著提升，准确率最高提升24%。
英文摘要：
In an effort to automatically evaluate and select the best model and improve code quality for automatic incident remediation in IT Automation, it is crucial to verify if the generated code for remediation action is syntactically and semantically correct and whether it can be executed correctly as intended. There are three approaches: 1) conventional methods use surface form similarity metrics (token match, exact match, etc.) which have numerous limitations, 2) execution-based evaluation focuses more on code functionality based on pass/fail judgments for given test-cases, and 3) LLM-as-a-Judge employs LLMs for automated evaluation to judge if it is a correct answer for a given problem based on pre-defined metrics. In this work, we focused on enhancing LLM-as-a-Judge using bidirectional functionality matching and logic representation for reference-less automatic validation and refinement for Bash code generation to select the best model for automatic incident remediation in IT Automation. We used execution-based evaluation as ground-truth to evaluate our LLM-as-a-Judge metrics. Results show high accuracy and agreement with execution-based evaluation (and up to 8% over baseline). Finally, we built Reflection code agents to utilize judgments and feedback from our evaluation metrics which achieved significant improvement (up to 24% increase in accuracy) for automatic code refinement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
