Context Matters: 一种用于科学教育的语言模型预训练策略
Context Matters: A Strategy to Pre-train Language Model for Science Education
AI总结:
本研究提出使用科学教育数据对BERT和SciBERT进行持续预训练,实验表明领域内学生作答语料能显著提升模型在科学论证自动评分等下游任务上的性能。
摘要:
本研究旨在提高自动评分科学教育中学生作答的准确性。基于BERT的语言模型在各种语言相关任务中显示出优于传统NLP模型的显著优势。然而,学生的科学写作(包括论证和解释)具有领域特定性。此外,学生使用的语言与BERT及其现有变体训练来源的期刊和维基百科中的语言不同。所有这些表明,使用科学教育数据预训练的领域特定模型可能会提高模型性能。然而,用于上下文化预训练语言模型并提高自动评分学生书面作答性能的理想数据类型仍不明确。因此,我们在本研究中采用不同数据来上下文化BERT和SciBERT模型,并比较它们在科学论证评估任务自动评分上的表现。我们使用三个数据集来预训练模型:1)科学教育领域的期刊文章,2)一个包含学生书面作答的大型数据集(样本量超过50,000),以及3)一个小型的科学论证任务学生书面作答数据集。我们的实验结果表明,由科学问题和作答构建的领域内训练语料库提高了语言模型在多种下游任务上的性能。我们的研究证实了在教育领域对领域特定数据进行持续预训练的有效性,并展示了一种以高准确率自动化科学教育任务的可推广策略。我们计划发布我们的数据和SciEdBERT模型供公众使用和社区参与。
英文摘要:
This study aims at improving the performance of scoring student responses in science education automatically. BERT-based language models have shown significant superiority over traditional NLP models in various language-related tasks. However, science writing of students, including argumentation and explanation, is domain-specific. In addition, the language used by students is different from the language in journals and Wikipedia, which are training sources of BERT and its existing variants. All these suggest that a domain-specific model pre-trained using science education data may improve model performance. However, the ideal type of data to contextualize pre-trained language model and improve the performance in automatically scoring student written responses remains unclear. Therefore, we employ different data in this study to contextualize both BERT and SciBERT models and compare their performance on automatic scoring of assessment tasks for scientific argumentation. We use three datasets to pre-train the model: 1) journal articles in science education, 2) a large dataset of students' written responses (sample size over 50,000), and 3) a small dataset of students' written responses of scientific argumentation tasks. Our experimental results show that in-domain training corpora constructed from science questions and responses improve language model performance on a wide variety of downstream tasks. Our study confirms the effectiveness of continual pre-training on domain-specific data in the education domain and demonstrates a generalizable strategy for automating science education tasks with high accuracy. We plan to release our data and SciEdBERT models for public use and community engagement.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。