基于大语言模型的阅读理解测试题自动生成与评估
Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models
AI总结：
本研究构建德语阅读理解题目数据集，提出含文本信息性指标的评估方案，验证了大语言模型可零样本生成和评估阅读理解题，且GPT-4表现优于Llama 2，为低资源语言的试题生成提供了可行路径。
摘要：
阅读理解测试被应用于从教育到简化文本可读性评估的多种场景中。然而，人工创建这类测试并确保其质量难度大且耗时。本文探究了大语言模型（LLMs）可如何用于生成和评估多项选择题型的阅读理解题目。为此，我们构建了一个德语阅读理解题目数据集，并制定了一套新的人工与自动评估方案，其中包含一个我们称之为文本信息性的指标，该指标基于可猜测性与可回答性构建。随后我们使用该方案和数据集评估了Llama 2与GPT-4生成的题目质量。结果表明，两个模型都能在零样本设置下生成质量可接受的题目，但GPT-4的表现明显优于Llama 2。我们还证明，通过引导大语言模型给出题目作答，可将其用于自动评估。在该场景下，GPT-4的评估结果与人类标注者的结果最为相似。总体而言，利用大语言模型进行零样本生成是生成和评估阅读理解测试题的一种有前景的方法，尤其适用于缺乏大量可用数据的语言。
英文摘要：
Reading comprehension tests are used in a variety of applications, reaching from education to assessing the comprehensibility of simplified texts. However, creating such tests manually and ensuring their quality is difficult and time-consuming. In this paper, we explore how large language models (LLMs) can be used to generate and evaluate multiple-choice reading comprehension items. To this end, we compiled a dataset of German reading comprehension items and developed a new protocol for human and automatic evaluation, including a metric we call text informativity, which is based on guessability and answerability. We then used this protocol and the dataset to evaluate the quality of items generated by Llama 2 and GPT-4. Our results suggest that both models are capable of generating items of acceptable quality in a zero-shot setting, but GPT-4 clearly outperforms Llama 2. We also show that LLMs can be used for automatic evaluation by eliciting item reponses from them. In this scenario, evaluation results with GPT-4 were the most similar to human annotators. Overall, zero-shot generation with LLMs is a promising approach for generating and evaluating reading comprehension test items, in particular for languages without large amounts of available data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
