Qworld: Question-Specific Evaluation Criteria for LLMs
Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Medicine, Brigham and Women’s Hospital(布里洛妇产科医院医学部);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究 institute);Broad Institute of MIT and Harvard(MIT 和哈佛大学Broad研究所);Harvard Data Science Initiative(哈佛大学数据科学计划)
Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik
评估大型语言模型(LLMs)在开放性问题上的表现具有挑战性,因为响应质量依赖于问题的上下文。二元评分和静态评分标准无法捕捉这些上下文依赖性要求。现有方法在数据集层面定义标准或单次生成标准,限制了探索每个问题隐含的评估空间能力。我们引入One-Question-One-World(Qworld),一种通过递归扩展树生成问题特定评估标准的方法。给定一个问题,Qworld通过结构化的层次和横向扩展分解为场景、视角和细粒度二元标准。生成的标准指明高质量答案必须针对该问题解决的内容。在HealthBench上,Qworld覆盖89%的专家制定标准并生成79%的新标准,经人类专家验证。专家对Qworld标准的洞察力和细粒度评分高于先前方法。当应用于HealthBench和Humanity's Last Exam上的11个前沿LLM时,Qworld揭示了长期影响、公平性、错误处理和跨学科推理等维度的能力差异,这些差异粗略评分标准无法区分。通过将标准生成视为问题隐含评估轴的结构化覆盖,Qworld实现了针对每个问题的评估,而非依赖固定任务级标准。
英文摘要:
Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not capture. By generating evaluation criteria for each question, Qworld enables assessment of LLM responses that is tailored to the question rather than based on fixed task-level criteria.
LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing
Department of Linguistics & Institute for Advanced Computational Science, Stony Brook University(语言学系与高级计算科学研究所,石溪大学);Department of Linguistics, University of Saskatchewan(语言学系,萨斯喀彻温大学)
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Renmin University of China(中国人民大学);Xiaohongshu(小红书);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Shanghai Jiaotong University(上海交通大学);Peking University(北京大学)
Rensselaer Polytechnic Institute(伦斯勒理工学院);IBM Research(IBM研究院);University of Waterloo(滑铁卢大学);University of Michigan(密歇根大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria
POSTECH;Seoul National University(首尔国立大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Kyungpook National University(庆北国立大学);LG AI Research(LG AI研究院)
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心);University of Alicante (UA)(阿利坎特大学(UA));GOI Institute, Basque Summer University (UEU)(巴斯克夏季大学(UEU)GOI研究所)