arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09030cs.CL
收藏

通过反思-修订实现语言模型作文评分量规的自动化优化

Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise

The University of Tokyo(东京大学)

Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

AI总结:

本研究针对大型语言模型在自动化作文评分中对提示敏感的问题,提出一种通过反思模型自身评分理由及与人类评分差异来迭代优化评分量规的方法,实验证明其能显著提升评分一致性,即使使用简单初始量规也能达到与人类撰写量规相当的效果。

摘要:

大型语言模型(LLMs)的性能对所提供的提示高度敏感。受提示优化领域的启发,本研究探讨了通过让LLM优化评分量规来提升自动化作文评分(AES)的潜力。具体而言,该方法促使模型通过反思自身的评分理由及其在样本作文上与人类评分的差异,来迭代优化量规。 在TOEFL11和ASAP数据集上使用GPT-4o、Gemini-2.5-Pro和Qwen-3-32B-A3B-Instruct的实验表明,二次加权Kappa(QWK)得分分别提升了0.19和0.47。 值得注意的是,即使使用简单的初始量规,该方法也能取得与使用详细的人类撰写量规相当或更好的QWK得分。 研究结果突显了迭代优化量规对LLM在AES中与人类评价对齐的重要性。

英文摘要:

The performance of Large Language Models (LLMs) is highly sensitive to the prompts they are given. Drawing inspiration from the field of prompt optimization, this study investigates the potential for enhancing Automated Essay Scoring (AES) by refining the scoring rubrics used by LLMs. Specifically, our approach prompts models to iteratively refine rubrics by reflecting on models' own scoring rationales and observed discrepancies with human scores on sample essays. Experiments on the TOEFL11 and ASAP datasets using GPT-4.1, Gemini-2.5-Pro, and Qwen-3-Next-80B-A3B-Instruct show Quadratic Weighted Kappa (QWK) improvements of up to 0.19 and 0.47, respectively. Notably, even with a simple initial rubric, our approach achieves comparable or better QWK than using detailed human-authored rubrics. Our findings highlight the importance of iterative rubric refinement in LLM-based AES to enhance alignment with human evaluations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑