arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.23712cs.CLcs.AI
收藏

STED与一致性评分:一个评估LLM结构化输出可靠性的框架

STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability

AWS Generative AI Innovation Center(AWS生成式AI创新中心);AWS WWSO SA Field Initiatives(AWS WWSO SA现场项目)

Guanghui Wang, Jinze Yu, Xing Zhang, Dayuan Jiang, Yin Song, Tomal Deb, Xuefeng Liu, Peiyang He

AI总结:

本文提出STED与一致性评分框架,用于评估LLM结构化输出可靠性,通过实验展示其在不同模型上的表现差异及应用价值。

摘要:

大型语言模型(LLMs)越来越多地用于结构化数据生成,但输出一致性对于生产应用至关重要。我们介绍了一个全面的框架,用于评估和提高LLM生成结构化输出的一致性。我们的方法结合了:(1)STED(语义树编辑距离),一种新颖的相似性度量标准,在比较JSON输出时平衡语义灵活性与结构严格性;(2)一致性评分框架,通过多次生成的STED测量聚合,以量化可靠性。通过系统地在受控模式、表达和语义变化的合成数据集上进行实验,我们证明STED在语义等价物上表现优异(相似度0.86-0.90),在结构断裂时为0,优于现有的TED、BERTScore和DeepDiff等指标。将我们的框架应用于六个基准LLM,揭示了显著的差异:Claude-3.7-Sonnet表现出极高的一致性,即使在高温(T=0.9)下也能保持几乎完美的结构可靠性,而Claude-3-Haiku和Nova-Pro等模型则表现出显著的退化,需要仔细调整。我们的框架使实际应用成为可能,包括针对结构任务的定向模型选择、迭代提示优化以获得可重复的结果,以及诊断分析以识别不一致的根本原因。这项工作为确保LLM生产系统中的可靠结构化输出生成提供了理论基础和实用工具。

英文摘要:

Large Language Models (LLMs) are increasingly deployed for structured data generation, yet output consistency remains critical for production applications. We introduce a comprehensive framework for evaluating and improving consistency in LLM-generated structured outputs. Our approach combines: (1) STED (Semantic Tree Edit Distance), a novel similarity metric balancing semantic flexibility with structural strictness when comparing JSON outputs, and (2) a consistency scoring framework aggregating multiple STED measurements across repeated generations to quantify reliability. Through systematic experiments on synthetic datasets with controlled schema, expression, and semantic variations, we demonstrate STED achieves superior performance ($0.86-0.90$ similarity for semantic equivalents, $0.0$ for structural breaks) compared to existing metrics including TED, BERTScore, and DeepDiff. Applying our framework to benchmark six LLMs reveals significant variations: Claude-3.7-Sonnet demonstrates exceptional consistency, maintaining near-perfect structural reliability even at high temperatures ($T=0.9$), while models like Claude-3-Haiku and Nova-Pro exhibit substantial degradation requiring careful tuning. Our framework enables practical applications including targeted model selection for structured tasks, iterative prompt refinement for reproducible results, and diagnostic analysis to identify inconsistency root causes. This work provides theoretical foundations and practical tools for ensuring reliable structured output generation in LLM-based production systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑