arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23522cs.CLcs.AI
收藏

Qworld: 为LLMs设计的问题特定评估标准

Qworld: Question-Specific Evaluation Criteria for LLMs

Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Medicine, Brigham and Women’s Hospital(布里洛妇产科医院医学部);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究 institute);Broad Institute of MIT and Harvard(MIT 和哈佛大学Broad研究所);Harvard Data Science Initiative(哈佛大学数据科学计划)

Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik

AI总结:

Qworld通过递归扩展树生成问题特定评估标准,覆盖89%专家标准并产生79%新标准,揭示LLM在长期影响、公平性等维度的能力差异。

摘要:

评估大型语言模型(LLMs)在开放性问题上的表现具有挑战性,因为响应质量依赖于问题的上下文。二元评分和静态评分标准无法捕捉这些上下文依赖性要求。现有方法在数据集层面定义标准或单次生成标准,限制了探索每个问题隐含的评估空间能力。我们引入One-Question-One-World(Qworld),一种通过递归扩展树生成问题特定评估标准的方法。给定一个问题,Qworld通过结构化的层次和横向扩展分解为场景、视角和细粒度二元标准。生成的标准指明高质量答案必须针对该问题解决的内容。在HealthBench上,Qworld覆盖89%的专家制定标准并生成79%的新标准,经人类专家验证。专家对Qworld标准的洞察力和细粒度评分高于先前方法。当应用于HealthBench和Humanity's Last Exam上的11个前沿LLM时,Qworld揭示了长期影响、公平性、错误处理和跨学科推理等维度的能力差异,这些差异粗略评分标准无法区分。通过将标准生成视为问题隐含评估轴的结构化覆盖,Qworld实现了针对每个问题的评估,而非依赖固定任务级标准。

英文摘要:

Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not capture. By generating evaluation criteria for each question, Qworld enables assessment of LLM responses that is tailored to the question rather than based on fixed task-level criteria.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑