From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);KE Holdings Inc.(贝壳控股有限公司);School of Information, Central University of Finance and Economics(中央财经大学信息学院)
Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai
Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Qwen Large Model Application Team, Alibaba(阿里巴巴文勤大模型应用团队);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Monotonic Reference-Free Refinement for Autoformalization
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK Manchester Institute(国家生物标记中心,CRUK曼彻斯特研究所)
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving
University of Manchester(曼彻斯特大学);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK-MI, University of Manchester(英国曼彻斯特大学英国癌症研究中心曼彻斯特研究所国家生物标志物中心)
RubRIX: Rubric-Driven Risk Mitigation in Caregiver-AI Interactions
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);OSF HealthCare(OSF医疗集团);Indiana University Indianapolis(印第安纳大学印第安纳波利斯分校)
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University of Technology(浙江工业大学)
Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement
University of Manchester(曼彻斯特大学);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK-MI, University of Manchester(曼彻斯特大学英国癌症研究中心曼彻斯特研究所国家生物标志物中心)
PEIRCE: Unifying Material and Formal Reasoning via LLM-Driven Neuro-Symbolic Refinement
University of Manchester(曼彻斯特大学);Idiap Research Institute(伊迪亚普研究所);National Biomarker Centre, CRUK-MI, University of Manchester(英国癌症研究中心曼彻斯特研究所国家生物标志物中心,曼彻斯特大学);University of Galway(戈尔韦大学)