AI-TA：基于开源大语言模型的智能问答教学助教研究
AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs
AI总结：
针对在线课程问答人力成本高的问题，提出基于LLaMA-2开源大模型、融合RAG等技术的教育智能问答助教方案，在Piazza数据集上实现答案质量30%提升，为定制化AI教学助教开发铺路。
摘要：
每学期在线问答平台上数千条学生问题的回复需要耗费大量人力成本，在选课人数快速增长的计算机类课程中尤为突出。为解决可扩展智能问答（QA）的相关挑战，我们提出了一种创新方案，采用LLaMA-2系列的开源大语言模型（LLMs）以保障数据隐私。该方法融合了检索增强生成（RAG）、监督微调（SFT）等增强技术，还通过直接偏好优化（DPO）从人类偏好数据中学习。我们在一门计算机入门课程的Piazza数据集上开展了大量实验，该数据集包含10000个问答对和1500对偏好数据，结果显示答案质量显著提升30%，其中RAG的加入效果尤为突出。我们的贡献包括：构建了一种面向教育场景问答的新型架构，结合人类评估与基于LLM的指标对LLM性能进行了全面评测，同时梳理了教育数据处理的挑战与未来方向。本工作为AI-TA的开发奠定了基础，这是一款可针对设有在线问答平台的课程进行定制的智能问答助教。
英文摘要：
Responding to the thousands of student questions on online QA platforms each semester has a considerable human cost, particularly in computing courses with rapidly growing enrollments. To address the challenges of scalable and intelligent question-answering (QA), we introduce an innovative solution that leverages open-source Large Language Models (LLMs) from the LLaMA-2 family to ensure data privacy. Our approach combines augmentation techniques such as retrieval augmented generation (RAG), supervised fine-tuning (SFT), and learning from human preferences data using Direct Preference Optimization (DPO). Through extensive experimentation on a Piazza dataset from an introductory CS course, comprising 10,000 QA pairs and 1,500 pairs of preference data, we demonstrate a significant 30% improvement in the quality of answers, with RAG being a particularly impactful addition. Our contributions include the development of a novel architecture for educational QA, extensive evaluations of LLM performance utilizing both human assessments and LLM-based metrics, and insights into the challenges and future directions of educational data processing. This work paves the way for the development of AI-TA, an intelligent QA assistant customizable for courses with an online QA platform
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
