arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10017cs.CL
收藏

SCORE:特定性、上下文利用、鲁棒性与相关性用于无参考LLM评估

SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation

University of Illinois Chicago(伊利诺伊大学芝加哥分校);Argonne National Laboratory(阿贡国家实验室);University of Pennsylvania(宾夕法尼亚大学)

Homaira Huda Shomee, Rochana Chaturvedi, Yangxinyu Xie, Tanwi Mallick

AI总结:

本文提出了一种无参考评估框架,用于评估LLM在高风险领域任务中的特定性、鲁棒性、相关性和上下文利用,通过精心编纂的数据集和人工评估验证了多指标评估的必要性。

摘要:

大型语言模型(LLM)日益被用于支持在高风险、领域特定的环境中进行问答和决策,如自然灾害响应和基础设施规划,其中有效的答案必须传达精细的、决策关键的细节。然而,现有的检索增强生成(RAG)和开放式问答评估框架主要依赖于表面相似性、事实一致性或语义相关性,往往无法评估响应是否提供领域敏感决策所需的具体信息。为解决这一差距,我们提出了一种多维、无参考评估框架,该框架沿四个互补维度评估LLM输出:特定性、对改写和语义扰动的鲁棒性、答案相关性和上下文利用。我们引入了一个包含1,412个领域特定问题-答案对的精心编纂数据集,涵盖40种专业角色和七种自然灾害类型,以支持系统性评估。我们进一步进行了人工评估,以评估标注者之间的一致性以及模型输出与人类判断之间的契合度,这突显了开放式、领域特定评估的固有主观性。我们的结果表明,单一指标无法单独捕捉答案质量,并展示了在高风险应用中部署LLM时,需要结构化、多指标评估框架的必要性。

英文摘要:

Large language models (LLMs) are increasingly used to support question answering and decision-making in high-stakes, domain-specific settings such as natural hazard response and infrastructure planning, where effective answers must convey fine-grained, decision-critical details. However, existing evaluation frameworks for retrieval-augmented generation (RAG) and open-ended question answering primarily rely on surface-level similarity, factual consistency, or semantic relevance, and often fail to assess whether responses provide the specific information required for domain-sensitive decisions. To address this gap, we propose a multi-dimensional, reference-free evaluation framework that assesses LLM outputs along four complementary dimensions: specificity, robustness to paraphrasing and semantic perturbations, answer relevance, and context utilization. We introduce a curated dataset of 1,412 domain-specific question-answer pairs spanning 40 professional roles and seven natural hazard types to support systematic evaluation. We further conduct human evaluation to assess inter-annotator agreement and alignment between model outputs and human judgments, which highlights the inherent subjectivity of open-ended, domain-specific evaluation. Our results show that no single metric sufficiently captures answer quality in isolation and demonstrate the need for structured, multi-metric evaluation frameworks when deploying LLMs in high-stakes applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑