iRULER: 可理解的基于评分标准的用户自定义LLM评估用于修改
iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision
AI总结：
iRULER通过特定评分标准和可解释反馈，提升写作审查和修改的可理解性和有效性。
摘要：
大型语言模型（LLMs）已成为评估写作不可或缺的工具。然而，它们提供的文本反馈往往不可理解、通用且不针对用户标准。受教育中的结构化评分标准和可理解的AI解释启发，我们根据识别的设计指南提出iRULER，通过特定标准来指导审查过程，提供评分选择的解释，并提供可操作的修改以针对不同的质量水平。为了资格用户自定义标准，我们递归地使用iRULER与评分标准的评分标准来迭代地细化评分标准。在受控实验中，iRULER在写作修改和评分标准创建中对验证的LLM判断审查评分改善最多，并被感知为最有助于人和与之对齐，相比只读评分标准和基于文本的LLM反馈。定性发现进一步支持iRULER如何满足用户自定义反馈的设计指南。这项工作为可理解的基于LLM的写作审查和修改提供了交互式评分标准工具，并且为用户自定义评分标准创建做出了贡献。
英文摘要：
Large Language Models (LLMs) have become indispensable for evaluating writing. However, text feedback they provide is often unintelligible, generic, and not specific to user criteria. Inspired by structured rubrics in education and intelligible AI explanations, we propose iRULER following identified design guidelines to \textit{scaffold} the review process by \textit{specific} criteria, providing \textit{justification} for score selection, and offering \textit{actionable} revisions to target different quality levels. To \textit{qualify} user-defined criteria, we recursively used iRULER with a rubric-of-rubrics to iteratively \textit{refine} rubrics. In controlled experiments on writing revision and rubric creation, iRULER most improved validated LLM-judged review scores and was perceived as most helpful and aligned compared to read-only rubric and text-based LLM feedback. Qualitative findings further support how iRULER satisfies the design guidelines for user-defined feedback. This work contributes interactive rubric tools for intelligible LLM-based review and revision of writing, and user-defined rubric creation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
