发表机构
NVIDIA（英伟达）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究探讨硬件设计验证中LLM自动进化测试平台的效果，发现收益难以跨任务整合，建议采用存档感知选择以利用互补专业化。
AI 中文摘要
智能体行为取决于语言模型周围的测试平台，但目前尚不清楚语言模型能否可靠地改进此类用于硬件设计任务的测试平台。我们在12个专有设计验证根因定位任务上，围绕固定主体模型研究了自动测试平台进化。每个任务进行五次试验，自动进化的测试平台使完成尝试次数增加了71-76%，任意命中任务覆盖率增加了80-100%，而总正确尝试次数仅提高了18-24%。至少重复两次的最强成功结果使一个任务得到改进，后续候选者在任务间交换收益而非保留收益。一个辅助候选者在排除在搜索之外的四个任务验证集上有所改进，但在随后包含搜索和验证任务的12个任务重放中与基线持平，因此所选收益并未在整个任务池中持续。在测试的血统中，有用的搜索、证据和最终化行为出现在不同候选者中，但未一致地整合到一个在任务和指标上均占主导地位的单一测试平台中。在另一项CVDP跨基准案例研究中，自动进化的定义宽度修复测试平台比其142任务参考基线多产生35.6%的功能通过；最终功能验证器对完成输出进行评分，但在修复过程中未向主体智能体展示。这些结果支持在进化产生互补专业化且缺乏一致整合时，采用存档感知选择。
英文摘要
Agent behavior depends on the harness surrounding a language model, but it remains unclear whether language models can reliably improve such harnesses for hardware-design tasks. We study automatic harness evolution around a fixed subject model on 12 proprietary design-verification root-cause localization tasks. Across five trials per task, automatically evolved harnesses increased completed attempts by 71-76% and any-hit task coverage by 80-100%, while total correct attempts improved by only 18-24%. The strongest success reproducible at least twice result improved by one task, and later candidates exchanged gains across tasks rather than preserving them. An auxiliary candidate improved on a four-task validation set excluded from search but tied its baseline on a subsequent 12-task replay containing both search and validation tasks, so the selected gain did not persist across the full pool. Across the tested lineage, useful search, evidence, and finalization behaviors appeared in different candidates but did not consistently consolidate into a single harness that dominated across tasks and metrics. In a separate CVDP cross-benchmark case study, an automatically evolved defined-width repair harness produced 35.6% more functional passes than its 142-task reference baseline; the final functional verifier scored completed outputs but was not shown to the subject agent during repair. These results support archive-aware selection when evolution yields complementary specializations without consistent consolidation.
