CodingTeachLLM:通过AST先验知识增强LLM的编程能力
CodingTeachLLM: Empowering LLM's Coding Ability via AST Prior Knowledge
AI总结:
本文提出专为编程教学设计的CodingTeachLLM模型,通过基于AST先验的三阶段监督微调实现教育知识的结构化拆解与增量引导输出,在HumanEval基准上达75.10%,同时保持强大的对话能力。
摘要:
本文介绍了CodingTeachLLM,一个专为编程教学设计的大型语言模型(LLM)。具体而言,我们旨在增强LLM的编程能力,并引导其在教育情境中进入更好的教学模式。因此,我们提出了一个端到端的基于先验的三阶段监督微调模型,该模型被证明比传统微调方法更具竞争力。更具体地说,我们的模型实现了教育知识的结构化拆解和增量引导输出。为此,我们通过采样器和重叠估计神经网络强化了三类的数据分类,并将预处理数据集分三批注入预训练模型进行LORA微调。然后,我们设计了一个先验模块,将系统提示、向量数据库和抽象语法树(AST)任务分割耦合起来。最后,将压缩方法和正则化约束应用于基于先验的微调模型,随后在输出端进行文本过滤以获得增量引导结果。我们的模型代表了首个真正体现导师角色的研究努力,具有丰富的教育知识、逐步增量引导输出和不披露答案的特征。大量实验报告表明,与开源模型相比,我们的模型在编程能力上也达到了最先进水平,在HumanEval(@pass 1)基准上达到了令人印象深刻的75.10%。此外,我们的模型保持了强大的对话能力,其13B量化版本在MMLU、C-Eval和AGIEval(5 shot)对话评估基准上分别取得了56.34、50.60和45.27的分数。
英文摘要:
In this paper, we introduce CodingTeachLLM, a large language model (LLM) designed for coding teaching. Specially, we aim to enhance the coding ability of LLM and lead it to better teaching mode in education context. Thus, we propose an end-to-end prior-based three-phases supervised fine-tuned model, which is proved more competitive than traditional fine-tuning method. More specifically, our model realizes the structural disassembly and incremental guided output of educational knowledge. To this end, we robustify data classification of three types via a sampler and overlap estimation neural network, and inject the preprocessing datasets into pre-trained model in three batches for LORA fine-tuning. Then, we design a prior module couples system prompt, vector databases, and abstract syntax tree task segmentation. Finally, the compression method and regularization constraint are applied to the prior-based fine-tuned model, followed by text filter at the output end to obtain incremental guided results. Our model represents the first research effort to truly embody the tutor role with the features of abundant educational knowledge, step-by-step incremental guided outputs and non-disclosure of answers. Extensive experiments report that our model also achieves state-of-the-art in code abilities compared to open-source models, reaching an impressive 75.10% on the HumanEval (@pass 1) benchmark. Additionally, our model maintains strong conversational capabilities, with the 13B quantized version achieving scores of 56.34, 50.60, and 45.27 respectively on the MMLU, C-Eval, and AGIEval (5 shot) dialogue evaluation benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。