Large Language Models (LLMs) are increasingly deployed for structured data generation, yet output consistency remains critical for production applications. We introduce a comprehensive framework for evaluating and improving consistency in LLM-generated structured outputs. Our approach combines: (1) STED (Semantic Tree Edit Distance), a novel similarity metric balancing semantic flexibility with structural strictness when comparing JSON outputs, and (2) a consistency scoring framework aggregating multiple STED measurements across repeated generations to quantify reliability. Through systematic experiments on synthetic datasets with controlled schema, expression, and semantic variations, we demonstrate STED achieves superior performance ($0.86-0.90$ similarity for semantic equivalents, $0.0$ for structural breaks) compared to existing metrics including TED, BERTScore, and DeepDiff. Applying our framework to benchmark six LLMs reveals significant variations: Claude-3.7-Sonnet demonstrates exceptional consistency, maintaining near-perfect structural reliability even at high temperatures ($T=0.9$), while models like Claude-3-Haiku and Nova-Pro exhibit substantial degradation requiring careful tuning. Our framework enables practical applications including targeted model selection for structured tasks, iterative prompt refinement for reproducible results, and diagnostic analysis to identify inconsistency root causes. This work provides theoretical foundations and practical tools for ensuring reliable structured output generation in LLM-based production systems.
From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Lund University(吕勒奥大学);University of Tübingen(图宾根大学);Stanford Graduate School of Education(斯坦福大学教育研究生院);Harvard Graduate School of Education(哈佛大学教育研究生院)
How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
The Chinese University of Hong Kong(香港中文大学);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tsinghua University(清华大学)
ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making
Kyoto University, Department of Biomedical Data Intelligence(京都大学生物医学数据智能系);Kyoto University, Department of Cardiovascular Medicine(京都大学心血管医学系);The University of Tokyo(东京大学)
InFoBench: Evaluating Instruction Following Ability in Large Language Models
Tencent AI Lab, Seattle(腾讯人工智能实验室(西雅图));University of Central Florida(中佛罗里达大学);University of Georgia(佐治亚大学);Emory University(埃默里大学);Shanghai Jiao Tong University(上海交通大学)
Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems
School of Computer Science and Engineering at Nanyang Technological University(南洋理工大学计算机科学与工程学院);School of Cyber Science and Engineering, Wuhan University(武汉大学网络科学与工程学院)
Budget-Sensitive Discovery Scoring: A Formally Verified Framework for Evaluating AI-Guided Scientific Selection
Indian Institute of Information Technology Allahabad (IIITA)(印度阿瓦德理工学院信息学院(IIITA));National Institute of Electronics & Information Technology (NIELIT)(国家电子与信息科技研究所(NIELIT))
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Tongji University(同济大学);AIR, Tsinghua University(清华大学人工智能研究院);BAAI(百度人工智能研究院)
From Diagnosis to Redesign: Using Quantitative Ethnography to Improve Multi-Agent LLM Reasoning
University of California, Irvine(加利福尼亚大学欧文分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Monash University(莫纳什大学);Columbia University(哥伦比亚大学);University of Pennsylvania(宾夕法尼亚大学)
Using Large Language Models to Assess Teachers' Pedagogical Content Knowledge
AI4STEM Education Center, University of Georgia, Athens, GA 30602, USA(AI4STEM教育中心,佐治亚大学,亚特兰大,GA 30602,USA);Department of Educational Psychology, University of Georgia, Athens, GA 30602, USA(教育心理学系,佐治亚大学,亚特兰大,GA 30602,USA);Department of Mathematics, Science, and Social Studies Education, University of Georgia, Athens, GA 30602, USA(数学、科学与社会科学教育系,佐治亚大学,亚特兰大,GA 30602,USA)
Evaluating LLM Personalization via Semantic Constraint Verification
University of New South Wales(新南威尔士大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);The Technology Innovation Institute(技术创新研究所);The Hong Kong University of Science and Technology(香港科技大学)
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
Southern University of Science and Technology(南方科技大学);University of Science and Technology of China(中国科学技术大学);University of Birmingham(伯明翰大学);Zhejiang University(浙江大学);East China Normal University(华东师范大学);Alibaba Group(阿里巴巴集团)
Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Indian Institute of Technology, Madras(印度理工学院马德拉斯分校);Robert Bosch Center for Data Sciences and AI (RBC-DSAI), Indian Institute of Technology, Madras(罗伯特博世数据科学与人工智能中心(RBC-DSAI),印度理工学院马德拉斯分校);Microsoft, India(微软印度公司);NVIDIA, India(英伟达印度公司)
自动评估非结构化领域的对话响应的质量是一个具有挑战性的问题。ADEM(Lowe et al. 2017)将对话系统的自动评估表述为一个学习问题,并表明这种模型能够预测与人类判断显著相关的反应,无论是在话语还是系统层面。他们的系统被证明击败了单词重叠指标,如BLEU,具有很大的利润率。我们从对手是否能利用ADEM模型的问题开始。针对基于神经网络的ADEM评估系统设计了一系列有针对性的攻击,表明对话系统的自动评估还有很长的路要走。ADEM可能会与文本中的词序颠倒这样简单的变体混淆!我们报告几个这样的对抗性的情况下,得出违反直觉的分数的对话反应的实验。我们采取了系统的看ADEM提出的评分功能,并将其连接到线性系统理论来预测系统中明显的缺点。我们还设计了一种攻击,可以欺骗这样的系统,以评价响应生成系统作为有利的。最后,我们提到未来的研究方向,使用对抗攻击,设计一个真正的自动化对话评估系统。
Automated Speech Scoring System Under The Lens: Evaluating and interpreting the linguistic cues for language proficiency
Columbia University(哥伦比亚大学);Second Language Testing, Inc.(第二语言测试公司);Indraprastha Institute of Information Technology, Delhi(德里印多尔信息科技学院);SUNY at Buffalo(布法罗纽约州立大学);Adobe Media Data Science Research(奥多比媒体数据科学研究中心)