arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02615cs.CRcs.AIcs.SE
收藏

代理创建,我们验证:用于代理工件生成的轻量级框架

TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation

Microsoft(微软)

Yaniv Melamed, Yoni Zukerman, Michal Shechter, Miri Weissler, Ashwin Patil, Hani Neuvirth-Telem

AI总结:

研究如何用大语言模型生成结构化工件并使其可靠用于生产部署。核心方法是基于“LLMs生成,我们验证”原则构建轻量级框架,贡献是提出含三关键属性的框架并在安全领域验证,还可用于其他工件生成任务。

摘要:

使用大语言模型生成结构化工件相对简单,但使其足够可靠用于生产部署存在挑战。我们提出一个基于核心原则的轻量级框架:大语言模型生成,我们验证。这种重新架构将责任从生成质量转移到验证严格性。该框架基于三个关键属性……我们在安全领域的三种工件类型上展示了该框架,相信其可应用于安全领域之外的其他工件生成任务。

英文摘要:

Generating structured artifacts with Large Language Models - e.g.\ database queries, threat framework mappings, entity schemas - is relatively straightforward; however, making them reliable enough for production deployments presents challenges. We present TAG, a lightweight framework based on a core principle: \textit{LLMs generate, we validate}. This reframing shifts responsibility from generation quality to validation rigor. The framework rests on three key attributes: First, \textbf{test driven generation}: when tests fail, the LLM receives indicative error messages that expose why the output failed, enabling the LLM to understand its mistakes and refine subsequent attempts. Second, \textbf{deterministic and LLM-based tests}: deterministic tests catch heuristics that can be programmatically verified (schema, syntax, cross-reference), while LLM-based tests evaluate nuanced semantic and delicate features that resist programmatic inspection (intent alignment, logical consistency, domain correctness). Third, \textbf{expert-distilled judges}: LLM-based tests are calibrated to distill and replicate human expert decision distribution, transforming manual human quality gates into scalable, reusable evaluation proxies that reflect professional-grade validation standards. We demonstrate the framework on three artifact types in the security domain - KQL query generation, MITRE ATT\&CK mapping, and entity mapping - deployed in production at Microsoft Sentinel. We believe this framework can be applied beyond security to other artifact generation tasks, providing a path to reliable, high-quality outputs without sacrificing the efficiency gains of LLM generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑