面向自动评分的日英句子翻译练习数据集
Japanese-English Sentence Translation Exercises Dataset for Automatic Grading
AI总结：
本研究提出第二语言学习早期阶段的句子翻译练习自动评分任务，构建含21道题、3498份作答的日英翻译练习数据集，实验显示微调BERT对正确作答分类F1约90%、错误作答不足80%，少样本GPT表现更差，该任务对大模型具挑战性。
摘要：
本文提出了句子翻译练习（Sentence Translation Exercises，STEs）的自动评估任务，这类练习常用于第二语言学习的早期阶段。我们将该任务形式化为：根据教育工作者预先指定的每条评分标准对学生作答进行评分。随后，我们构建了一个日英方向的句子翻译练习数据集，包含21道题目，总计3498份学生作答（平均每题167份），作答答案收集自学生与众包工作者。基于该数据集，我们测试了多个基线模型的性能，包括经过微调的BERT模型以及采用少样本上下文学习的GPT模型。实验结果显示，经过微调的BERT基线模型对正确作答的分类F1值约为90%，但对错误作答的分类F1值不足80%。此外，采用少样本学习的GPT模型表现比微调后的BERT更差，表明我们新提出的任务即使对最先进的大语言模型而言也颇具挑战性。
英文摘要：
This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs), that have been used in the early stage of L2 language learning. We formalize the task as grading student responses for each rubric criterion pre-specified by the educators. We then create a dataset for STE between Japanese and English including 21 questions, along with a total of 3, 498 student responses (167 on average). The answer responses were collected from students and crowd workers. Using this dataset, we demonstrate the performance of baselines including finetuned BERT and GPT models with few-shot in-context learning. Experimental results show that the baseline model with finetuned BERT was able to classify correct responses with approximately 90% in F1, but only less than 80% for incorrect responses. Furthermore, the GPT models with few-shot learning show poorer results than finetuned BERT, indicating that our newly proposed task presents a challenging issue, even for the stateof-the-art large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
