arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.01032cs.CLcs.AI
收藏

利用基于提示的技术为学校层级问题生成赋能:基于大型语言模型的方法

Harnessing the Power of Prompt-based Techniques for Generating School-Level Questions using Large Language Models

IIT Kharagpur(印度理工学院卡拉格普尔分校)

Subhankar Maity, Aniket Deroy, Sudeshna Sarkar

AI总结:

本文提出利用基于提示的技术和大型语言模型为学校层级科目自动生成高质量问题,构建了EduProbe数据集,并通过微调PEGASUS、T5、MBART、BART及评估GPT-3.5系列模型,发现T5在自动评估中表现最佳,但所有模型仍低于人类基线。

摘要:

设计高质量的教育问题是一项具有挑战性且耗时的工作。在这项工作中,我们提出了一种新颖的方法,利用基于提示的技术来生成描述性问题和基于推理的问题。然而,当前的问答(QA)数据集不足以支持我们在教育场景下开展基于提示的问题生成(QG)实验。因此,我们利用NCERT教科书的丰富内容,为学校层级科目构建了一个新的QG数据集,称为EduProbe。我们仔细地将该数据集标注为四元组,包括:1)上下文:构成问题所依据的文本片段;2)长提示:用于生成问题的较长文本线索(即覆盖上下文主要主题的较长单词或短语序列);3)短提示:用于生成问题的简短文本线索(即对上下文关键信息或重点的浓缩表示);4)问题:与上下文一致且与提示连贯的深层问题。我们通过微调预训练的基于Transformer的大型语言模型(LLMs),即PEGASUS、T5、MBART和BART,研究了多种基于提示的QG方法。此外,我们还探索了两个通用预训练LLM——Text-Davinci-003和GPT-3.5-Turbo——在无需进一步训练情况下的性能。通过自动评估,我们表明T5(使用长提示)优于所有其他模型,但仍未达到人类基线水平。在人类评估标准下,Text-Davinci-003在各种提示设置下通常表现出比其他模型更好的结果。即使在人类评估标准下,QG模型大多仍低于人类基线。我们的代码和数据集可在以下地址获取:https://github.com/my625/PromptQG

英文摘要:

Designing high-quality educational questions is a challenging and time-consuming task. In this work, we propose a novel approach that utilizes prompt-based techniques to generate descriptive and reasoning-based questions. However, current question-answering (QA) datasets are inadequate for conducting our experiments on prompt-based question generation (QG) in an educational setting. Therefore, we curate a new QG dataset called EduProbe for school-level subjects, by leveraging the rich content of NCERT textbooks. We carefully annotate this dataset as quadruples of 1) Context: a segment upon which the question is formed; 2) Long Prompt: a long textual cue for the question (i.e., a longer sequence of words or phrases, covering the main theme of the context); 3) Short Prompt: a short textual cue for the question (i.e., a condensed representation of the key information or focus of the context); 4) Question: a deep question that aligns with the context and is coherent with the prompts. We investigate several prompt-based QG methods by fine-tuning pre-trained transformer-based large language models (LLMs), namely PEGASUS, T5, MBART, and BART. Moreover, we explore the performance of two general-purpose pre-trained LLMs such as Text-Davinci-003 and GPT-3.5-Turbo without any further training. By performing automatic evaluation, we show that T5 (with long prompt) outperforms all other models, but still falls short of the human baseline. Under human evaluation criteria, TextDavinci-003 usually shows better results than other models under various prompt settings. Even in the case of human evaluation criteria, QG models mostly fall short of the human baseline. Our code and dataset are available at: https://github.com/my625/PromptQG

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑