arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.17022cs.CL
收藏

语言模型能否评估人类书写的文本?以教育领域的韩国学生写作为案例研究

Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education

Sangmoon High School(文昌高中);Carnegie Mellon University(卡内基梅隆大学)

Seungyoon Kim, Seungone Kim

AI总结:

本文研究 LLM 能否评估人类书写文本,收集 32 名韩国学生 100 篇文本并使用 GPT-4-Turbo 评估,发现其在语法性、流畅性及客观写作类型上可靠,但在其他方面表现不佳。

摘要:

大型语言模型(LLM)的评估流程已证明其能够稳健地评估机器生成的文本。将此方法扩展至评估人类书写的文本,可通过提供直接反馈以提升写作技能,从而显著惠及教育场景,尽管这一应用并非易事。本文研究 LLM 能否出于教育目的有效评估人类书写的文本。我们收集了 32 名韩国学生的 100 篇文本,涵盖 15 种写作类型,并采用 GPT-4-Turbo 以语法性、流畅性、连贯性、一致性和相关性为标准进行评估。分析表明,LLM 评估器能可靠地评估语法性、流畅性以及更客观的写作类型,但在其他标准和写作类型上表现不佳。我们公开了数据集和反馈。

英文摘要:

Large language model (LLM)-based evaluation pipelines have demonstrated their capability to robustly evaluate machine-generated text. Extending this methodology to assess human-written text could significantly benefit educational settings by providing direct feedback to enhance writing skills, although this application is not straightforward. In this paper, we investigate whether LLMs can effectively assess human-written text for educational purposes. We collected 100 texts from 32 Korean students across 15 types of writing and employed GPT-4-Turbo to evaluate them using grammaticality, fluency, coherence, consistency, and relevance as criteria. Our analyses indicate that LLM evaluators can reliably assess grammaticality and fluency, as well as more objective types of writing, though they struggle with other criteria and types of writing. We publicly release our dataset and feedback.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑