arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03306cs.CLcs.AI
收藏

以标记为导向的对象表示法与JSON:一种平滑解码和约束解码生成的基准测试

Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation

Ivan Matveev

AI总结:

本文比较了TOON与JSON在生成任务中的表现,发现TOON在特定条件下更高效,但受限于提示开销,而JSON在简单结构中表现更优。

摘要:

最近提出的以标记为导向的对象表示法(TOON)旨在取代JSON作为传递结构化数据给大语言模型(LLM)的序列化格式,显著减少token使用。尽管在LLM理解方面表现出色,但缺乏与JSON生成的测试。虽然TOON语法从未出现在训练数据中,但其简单性暗示了一次性上下文学习可能支持准确生成。不可避免的提示开销可以是更短完成的可接受折中。为此,我们进行了基准测试,创建了几个测试案例,涉及结构复杂性、验证流程以及比较普通JSON生成与结构化输出(通过约束解码)JSON生成与TOON一次性上下文学习生成。JSON结构化输出被包括在内,以建立最低token预算的基准线,并为未来测试TOON约束解码推理强制的实验设定起点。关键发现:TOON在领域生成任务中显示出有希望的准确率/token消耗比,但这种优势通常被较短上下文中“提示税”(指导开销)所抵消。普通JSON生成在一次性和最终准确率上表现最佳,即使与约束解码结构化输出相比,其唯一显著优势是最低token使用作为整体准确性稍低的折中,以及某些模型的显著退化。值得注意的是,对于简单结构,这种“最低token使用”约束解码的表现甚至超过了TOON,暗示通过如xgrammar等框架强制TOON可能无法达到预期效果。此外,结果表明了一种扩展假设:TOON的真实效率潜力可能遵循非线性曲线,在特定点之后,累计语法节省抵消初始提示开销后才会显现。

英文摘要:

Recently presented Token-Oriented Object Notation (TOON) aims to replace JSON as a serialization format for passing structured data to LLMs with significantly reduced token usage. While showing solid accuracy in LLM comprehension, there is a lack of tests against JSON generation. Though never present in training data, TOON syntax is simple enough to suggest one-shot in-context learning could support accurate generation. The inevitable prompt overhead can be an acceptable trade-off for shorter completions. To test this, we conducted a benchmark creating several test cases with regard to structural complexity, a validation pipeline, and comparing plain JSON generation vs structured output (via constrained decoding) JSON generation vs TOON one-shot in-context learning generation. JSON structured output was included to establish a minimum token budget baseline and to set a starting point for future experiments testing TOON constrained decoding inference enforcement. Key findings: TOON shows promising accuracy/token consumption ratio for in-domain generation tasks, though this advantage is often reduced by the "prompt tax" of instructional overhead in shorter contexts. Plain JSON generation shows the best one-shot and final accuracy, even compared with constrained decoding structured output, where the only significant advantage is the lowest token usage as a trade-off for slightly decreased accuracy overall and significant degradation for some models. Notably, for simple structures, this "lowest token usage" of constrained decoding outperformed even TOON, hinting that TOON enforcing via frameworks such as xgrammar may not yield the desired results. Furthermore, the results suggest a scaling hypothesis: TOON's true efficiency potential likely follows a non-linear curve, shining only beyond a specific point where cumulative syntax savings amortize the initial prompt overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑