Generating structured artifacts with Large Language Models - e.g.\ database queries, threat framework mappings, entity schemas - is relatively straightforward; however, making them reliable enough for production deployments presents challenges. We present TAG, a lightweight framework based on a core principle: \textit{LLMs generate, we validate}. This reframing shifts responsibility from generation quality to validation rigor. The framework rests on three key attributes: First, \textbf{test driven generation}: when tests fail, the LLM receives indicative error messages that expose why the output failed, enabling the LLM to understand its mistakes and refine subsequent attempts. Second, \textbf{deterministic and LLM-based tests}: deterministic tests catch heuristics that can be programmatically verified (schema, syntax, cross-reference), while LLM-based tests evaluate nuanced semantic and delicate features that resist programmatic inspection (intent alignment, logical consistency, domain correctness). Third, \textbf{expert-distilled judges}: LLM-based tests are calibrated to distill and replicate human expert decision distribution, transforming manual human quality gates into scalable, reusable evaluation proxies that reflect professional-grade validation standards. We demonstrate the framework on three artifact types in the security domain - KQL query generation, MITRE ATT\&CK mapping, and entity mapping - deployed in production at Microsoft Sentinel. We believe this framework can be applied beyond security to other artifact generation tasks, providing a path to reliable, high-quality outputs without sacrificing the efficiency gains of LLM generation.
A Comprehensive Survey of Agent Distillation across Models, Artifacts, and Harnesses
Central South University(中南大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Hong Kong Polytechnic University(香港理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
QiYuan Lab(启元实验室);University of Electronic Science and Technology of China(电子科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Northeastern University(东北大学);Beihang University(北京航空航天大学);Tsinghua University(清华大学)
University of Houston–Clear Lake(休斯顿大学清水湖分校);Infodat International Inc.(信息数据国际公司);Cardinal Health(红衣主教健康公司);Fontus Labs Inc.(方图斯实验室公司);Stockton University(斯托克顿大学)
DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation
Chinese Information Processing Laboratory(中国信息处理实验室);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学)
Agentic Unlearning: When LLM Agent Meets Machine Unlearning
Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine(山东中医药大学中医人工智能中心);School of Software, Shandong University(山东大学软件学院);School of Medical Information Engineering, Shandong University of Traditional Chinese Medicine(山东中医药大学医学信息工程学院);Shandong Huazhi Talent Technology Co., Ltd.(山东华智人才科技有限公司)
Foundations for Investigations of Agentic AI Systems from the Forensic Analysis of OpenClaw
KASTEL Security Research Labs, Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院 KASTEL 安全研究实验室);Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE(弗劳恩霍夫通信、信息处理与人体工程学研究所 FKIE)
Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence
Laboratory for Atomistic and Molecular Mechanics(原子分子力学实验室);Department of Biological Engineering(生物工程系);Massachusetts Institute of Technology(麻省理工学院);Department of Civil and Environmental Engineering(土木与环境工程系);Department of Mechanical Engineering(机械工程系);Center for Computational Science and Engineering(计算科学与工程中心);Schwarzman College of Computing(施瓦茨曼计算学院)