arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.12014cs.AIcs.SE
收藏

LLMs是否准备好迎接TOON?在新型结构化输出格式中评估结构正确性与可持续性的权衡

Are LLMs Ready for TOON? Benchmarking Structural Correctness-Sustainability Trade-offs in Novel Structured Output Formats

University of Naples Federico II(那不勒斯费迪里奇二世大学);University of Bergamo(贝拉姆乔大学)

Elio Masciari, Vincenzo Moscato, Enea Vincenzo Napolitano, Gian Marco Orlando, Marco Perillo, Diego Russo

AI总结:

本文提出了一种考虑环境效率的评估框架,评估新型结构化输出格式TOON在结构正确性与可持续性之间的权衡,发现TOON在紧凑性和排放方面表现优异,但需依赖模型容量以提升正确性。

摘要:

大型语言模型(LLMs)越来越多地被要求为下游系统生成结构化的、机器可读的输出。尽管最近的基准测试主要集中在评估此类输出的结构正确性上,但不同输出格式的推理环境影响却大多被忽视。在本文中,我们主张结构化输出格式应不仅在正确性方面进行评估,还应考虑其环境效率。为此,我们引入了一个可持续性意识的评估框架,用于结构生成,该框架测量了令牌使用、生成时间和估计的碳排放。在该框架下,我们提出了一个环境意识的生成正确性评分(GCS_env),这是一个整合了结构正确性与碳意识效率的统一指标。使用该框架,我们系统地将新的TOON格式与已建立的表示(JSON、XML、YAML)在多个LLM上进行比较,这些LLM涵盖不同的架构和参数规模。我们的结果揭示了一种一致的权衡:TOON产生明显更紧凑的输出和更低的排放,但当模型缺乏原生支持时,结构正确性会降低。我们展示了增加模型容量可以减少这种差距,并且环境意识评分可以根据部署优先级改变格式排名。这突显了需要包含可持续性的基准测试,并提供了实证证据,表明紧凑的表示如TOON可以在大规模、碳意识的LLM部署中提供实际优势。

英文摘要:

Large Language Models (LLMs) are increasingly required to generate structured, machine-readable outputs for downstream systems. While recent benchmarks have focused on evaluating the structural correctness of such outputs, the environmental impact of inference for different output formats has largely been overlooked. In this paper, we argue that structured output formats should be assessed not only in terms of correctness, but also with respect to their environmental efficiency. To this end, we introduce a sustainability-aware evaluation framework for structured generation that measures token usage, generation time, and estimated carbon emissions. Within this framework, we propose the Environment-Aware Generation Correctness Score (GCS_env), a unified metric that integrates structural correctness with carbon-aware efficiency. Using this framework, we systematically benchmark the novel TOON format against established representations (JSON, XML, YAML) across multiple LLMs spanning different architectures and parameter scales. Our results reveal a consistent trade-off: TOON yields markedly more compact outputs and lower emissions, but lower structural correctness when models lack native support. We show that increased model capacity reduces this gap and that environment-aware scoring can shift format rankings depending on deployment priorities. highlighting the need for sustainability-inclusive benchmarking and provides empirical evidence that compact representations such as TOON can offer practical advantages in large-scale, carbon-conscious LLM deployments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑