arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04426cs.AI
收藏

XGrammar-2: 面向智能体LLM的高效动态结构化生成引擎

XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs

Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);Carnegie Mellon University, NVIDIA(卡内基梅隆大学,NVIDIA)

Linzhang Li, Yixin Dong, Guanjie Wang, Ziyi Xu, Alexander Jiang, Tianqi Chen

AI总结:

针对智能体LLM中动态结构化生成(如工具调用和响应协议)的挑战,提出XGrammar-2引擎,通过标签触发结构切换和跨语法子结构缓存实现高效编译与近零开销。

摘要:

现代LLM智能体越来越依赖动态结构化生成,例如工具调用和响应协议。与具有静态结构的传统结构化生成不同,这些工作负载在请求之间和请求内部都有变化,给现有引擎带来了新的挑战。我们提出了XGrammar-2,一种用于动态智能体工作负载的结构化生成引擎。我们的设计基于两个关键思想:对标签触发的结构切换的一流支持,以及跨具有不同输出结构的请求的细粒度重用。具体来说,XGrammar-2引入了TagDispatch用于动态结构调度,以及Cross-Grammar Cache用于跨语法的子结构级缓存重用。它通过基于Earley的自适应令牌掩码缓存、即时编译和重复状态压缩进一步提高了效率。实验表明,XGrammar-2的编译速度比先前的结构化生成引擎快6倍以上,并且在现代LLM服务系统中几乎为零的端到端开销。

英文摘要:

Modern LLM agents increasingly rely on dynamic structured generation, such as tool calling and response protocols. Unlike traditional structured generation with static structures, these workloads vary both across requests and within a request, posing new challenges to existing engines. We present XGrammar-2, a structured generation engine for dynamic agentic workloads. Our design is based on two key ideas: first-class support for tag-triggered structure switching, and fine-grained reuse across requests with different output structures. Concretely, XGrammar-2 introduces TagDispatch for dynamic structural dispatching and Cross-Grammar Cache for substructure-level cache reuse across grammars. It further improves efficiency with an Earley-based adaptive token mask cache, just-in-time compilation, and repetition state compression. Experiments show that XGrammar-2 achieves over 6x faster compilation than prior structured generation engines, and incurs near-zero end-to-end overhead in modern LLM serving systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑