使教师模型与学员偏好对齐以实现定制化训练数据生成
Aligning Teacher with Student Preferences for Tailored Training Data Generation
AI总结：
本文提出ARTE框架，通过收集学员偏好并对齐教师模型，为知识蒸馏生成定制化训练示例，实验证明其优于现有指令微调数据集，并验证了学员模型与对齐教师模型的泛化能力。
摘要：
大型语言模型（LLMs）在各种任务中作为副驾驶（copilots）已展现出显著潜力。在处理隐私敏感数据或延迟敏感任务时，需要在边缘设备上本地部署LLMs。此类设备的计算约束使得直接部署强大的大规模LLMs不切实际，因此需要将知识从大规模模型蒸馏到轻量级模型。已有大量工作致力于从LLMs中引出多样且高质量的训练示例，但鲜有关注基于学员偏好来对齐教师的教学内容，这类似于教育学中的“响应式教学”。因此，我们提出了ARTE，全称为Aligning TeacheR with StudenT PreferencEs，这是一个使教师模型与学员偏好对齐的框架，用于为知识蒸馏生成定制化训练示例。具体而言，我们首先从教师模型中引出草拟问题与推理过程，然后利用学员在上下文学习中的表现作为代理来收集学员对这些问题和推理过程的偏好，最后使教师模型与学员偏好对齐。最终，我们使用对齐后的教师模型重复第一步，为目标任务上的学员模型引出定制化训练示例。在学术基准上的大量实验表明，ARTE优于从强大LLMs蒸馏得到的现有指令微调数据集。此外，我们深入研究了ARTE的泛化性，包括微调后学员模型在推理能力上的泛化性，以及对齐后的教师模型在跨任务和跨学员生成定制化训练数据方面的泛化性。总之，我们的贡献在于提出了一个用于定制化训练示例生成的新框架，在实验中证明了其有效性，并研究了ARTE中学员模型与对齐后教师模型的泛化性。
英文摘要：
Large Language Models (LLMs) have shown significant promise as copilots in various tasks. Local deployment of LLMs on edge devices is necessary when handling privacy-sensitive data or latency-sensitive tasks. The computational constraints of such devices make direct deployment of powerful large-scale LLMs impractical, necessitating the Knowledge Distillation from large-scale models to lightweight models. Lots of work has been done to elicit diversity and quality training examples from LLMs, but little attention has been paid to aligning teacher instructional content based on student preferences, akin to "responsive teaching" in pedagogy. Thus, we propose ARTE, dubbed Aligning TeacheR with StudenT PreferencEs, a framework that aligns the teacher model with student preferences to generate tailored training examples for Knowledge Distillation. Specifically, we elicit draft questions and rationales from the teacher model, then collect student preferences on these questions and rationales using students' performance with in-context learning as a proxy, and finally align the teacher model with student preferences. In the end, we repeat the first step with the aligned teacher model to elicit tailored training examples for the student model on the target task. Extensive experiments on academic benchmarks demonstrate the superiority of ARTE over existing instruction-tuning datasets distilled from powerful LLMs. Moreover, we thoroughly investigate the generalization of ARTE, including the generalization of fine-tuned student models in reasoning ability and the generalization of aligned teacher models to generate tailored training data across tasks and students. In summary, our contributions lie in proposing a novel framework for tailored training example generation, demonstrating its efficacy in experiments, and investigating the generalization of both student & aligned teacher models in ARTE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
