arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.13375cs.CL
收藏

从结构到有效性:大语言模型评分规则的机制引导生成与优化

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);KE Holdings Inc.(贝壳控股有限公司);School of Information, Central University of Finance and Economics(中央财经大学信息学院)

Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai

AI总结:

本研究通过系统分析评分规则设计的核心因素,提出ARTS-Hybrid算法,该算法可提升评估性能与评分规则质量,且生成的评分规则能为模型优化提供有效信号。

摘要:

评分规则已被广泛用于在开放式任务上评估和优化语言模型。然而,现有工作主要聚焦于生成更好的评分规则,对评分规则设计有效性的核心因素理解有限。本研究从标准组织、标准操作化和分数聚合三个维度对评分规则设计开展系统机制分析,发现结构化组织可实现更全面的标准构建、仅语义规范无法确保有效的标准级判断、复杂的分数聚合仅能提供有限的额外增益。基于上述发现,我们提出自适应评分规则树搜索与混合优化算法(ARTS-Hybrid),该算法结合自适应树构建与以语义为先、响应校准的优化。在多个基准上的实验表明,ARTS-Hybrid可同时提升端到端评估性能和内在评分规则质量,进一步实验显示,除评估外,ARTS-Hybrid生成的评分规则还能为模型优化提供有效信号。

英文摘要:

Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑