基于标注指南的知识增强：提升大语言模型在教育文本分类中的性能
Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification
AI总结：
提出AGKA方法，利用GPT-4.0从标注指南提取知识并辅以典型示例，增强LLM在教育文本分类中的性能，在六个LEC数据集上验证，GPT-4.0和Llama 3 70B表现优异。
摘要：
各种机器学习方法在自动化分类教育文本以识别学习投入指标方面已获得显著普及，即学习投入分类（LEC）。LEC能够为人类学习过程提供全面的洞察，吸引了包括自然语言处理（NLP）、学习分析和教育数据挖掘在内的多个研究社区的浓厚兴趣。近年来，大型语言模型（LLMs），如ChatGPT，在各种NLP任务中表现出卓越的性能。然而，它们在LEC任务中的全面评估和改进方法尚未得到深入研究。在本研究中，我们提出了基于标注指南的知识增强（AGKA）方法以改进LLMs。AGKA采用GPT 4.0从标注指南中检索标签定义知识，然后应用随机欠采样器选择少量典型示例。随后，我们进行了LEC的系统评估基准测试，该基准包括六个LEC数据集，涵盖行为分类（问题和紧急程度）、情感分类（二元和认知情感）以及认知分类（观点和认知存在）。研究结果表明，AGKA能够增强未经微调的LLMs，特别是GPT 4.0和Llama 3 70B。带有AGKA少样本的GPT 4.0在简单二元分类数据集上优于全样本微调模型如BERT和RoBERTa。然而，GPT 4.0在需要深入理解复杂语义信息的多类任务中表现落后。值得注意的是，基于开源LLM的Llama 3 70B与AGKA的组合具有前景，因为其性能与带有AGKA的闭源GPT 4.0相当。此外，LLMs在多类分类中难以区分名称相似的标签。
英文摘要：
Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
