arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.01588cs.CLcs.AI
收藏

(WhyPHI)针对多项选择题问答微调 PHI-3:方法、结果与挑战

(WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

Cairo University(开罗大学)

Mohamed Hisham Abdellatif

AI总结:

本文通过微调 PHI-3 并在 TruthfulQA 上优化提示,使其 MCQ 问答准确率从 62% 提升至 90.8%,展示了高效模型在教育评估中的应用潜力。

摘要:

大型语言模型(LLMs)因其在理解和生成类人文本方面的出色能力,已成为各个领域的重要工具。准确回答多项选择题(MCQs)的能力在教育领域具有重要价值,尤其是在自动化辅导系统和评估平台中。然而,由于幻觉现象和提示不清晰,使 LLMs 有效处理 MCQ 任务仍然具有挑战性。本研究探索了微软的 PHI-3(Abdin 等人,2024)这一紧凑而高效的 LLM 在 MCQ 问答中的潜力。我们的贡献包括:在 TruthfulQA 数据集上对模型进行微调,设计优化的提示以提升模型性能,并使用困惑度以及准确率和 F1 分数等传统指标进行评估。结果表明,微调后 PHI-3.5 的 MCQ 处理能力显著提升,困惑度从 4.68 降至 2.27,准确率从 62% 提升至 90.8%。本研究强调了高效模型在自适应学习系统和教育评估中的重要性,为其更广泛地融入课堂铺平了道路,尤其是在备考、学生反馈和个性化学习等领域。

英文摘要:

Large Language Models (LLMs) have become essential tools across various domains due to their impressive capabilities in understanding and generating human-like text. The ability to accurately answer multiple-choice questions (MCQs) holds significant value in education, particularly in automated tutoring systems and assessment platforms. However, adapting LLMs to handle MCQ tasks effectively remains challenging due to the hallucinations and unclear prompts. This work explores the potential of Microsoft's PHI-3\cite{Abdin2024}, a compact yet efficient LLM, for MCQ answering. Our contributions include fine-tuning the model on the TruthfulQA dataset, designing optimized prompts to enhance model performance, and evaluating using perplexity and traditional metrics like accuracy and F1 score. Results show a remarkable improvement in PHI-3.5's MCQ handling post-fine-tuning, with perplexity decreasing from 4.68 to 2.27, and accuracy rising from 62\% to 90.8\%. This research underlines the importance of efficient models in adaptive learning systems and educational assessments, paving the way for broader integration into the classroom, particularly in fields like test preparation, student feedback, and personalized learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑