规范驱动迁移中AI生成数据库规范的防泄漏评估
Leakage-Resistant Evaluation of AI-Generated Database Specifications in Specification-Driven Migration
AI总结:
本研究提出防泄漏框架与基准LRFB,评估GPT-4o等3个LLM评判者对AI生成Oracle PL/SQL规范的评估能力,发现其平衡准确率优于多数基线,且能提供长度基线缺失的缺陷类别与证据关联信息,推动规范评估向类型化发现转变。
摘要:
本文提出了一种用于评估AI生成Oracle PL/SQL规范的防泄漏框架与基准(LRFB)。该基准包含494个带标签的源规范-目标规范对,涵盖7个维度下的19项质量指标;每项指标包含26条记录,分为开发子集与保留子集各13条,每个子集含3个干净规范与10个缺陷规范。真实值通过机械方式密封以防止推理,并通过金丝雀测试与标签置换测试进行验证。对3个大语言模型(LLM)评判者进行评估,每条记录运行3次,并与2个简单基线进行比较。在247条保留记录上,GPT-4o、DeepSeek V4 Pro与Claude Sonnet 4.5的平衡准确率分别达到0.87、0.89与0.81,而规范长度基线达到0.88。精确配对测试未发现任何评判者与该基线存在显著差异。不过,评判者提供了长度基线无法获取的信息:在84%-92%的违规记录中,标注的缺陷类别出现在结果中,且引用的证据与标注跨度的重叠率为92%-93%。因此,本研究将规范评估从二元问题检测重新定位为与证据关联的类型化发现及基准有效性控制。
英文摘要:
This paper presents a leakage-resistant framework and benchmark (LRFB) for measuring evaluators of AI-generated Oracle PL/SQL specifications. The benchmark contains 494 labelled source-specification pairs across 19 quality metrics grouped into seven dimensions; each metric contains 26 records divided into development and held-out subsets of 13 records each, with three clean and ten defective specifications per subset. Ground truth is mechanically sealed from inference and checked with canary and label-permutation tests. Three LLM judges were evaluated with three runs per record and compared against two trivial baselines. On 247 held-out records, GPT-4o, DeepSeek V4 Pro, and Claude Sonnet 4.5 reached balanced accuracies of 0.87, 0.89, and 0.81, respectively, while a specification-length baseline reached 0.88. Exact paired tests did not separate any judge from that baseline. The judges nevertheless provided information unavailable to length: the labelled defect category appeared among findings in 84%-92% of violated records, and quoted evidence overlapped the labelled span in 92%-93%. The study therefore reframes specification evaluation from binary issue detection toward evidence-linked, typed findings and benchmark-validity controls.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。