发表机构
Korea University; Yonsei University Mirae Campus; Soongsil University; Konkuk University（高丽大学; 延世大学未来校区; 崇实大学; 建国大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出方向性标签蒸馏，利用教师模型在已知方向评分候选答案，生成更准确的训练标签，显著提升学生模型在开放式查询上的准确性，并减少错误转移。
AI 中文摘要
知识蒸馏旨在将大型语言模型的事实知识迁移到较小的模型，以实现高效部署。然而，教师模型可能在一个方向上回忆出关系，但在相反方向上却无法生成答案。因此，从其生成的答案进行蒸馏可能会将这种方向性局限传播给学生。然而，同一个教师可以通过在其已知的方向上对关系进行评分来识别这样的答案。我们引入了方向性标签蒸馏，其中冻结的教师模型在该已知方向上对候选答案进行评分，得分最高的候选答案成为学生的训练目标。在关于父母及其子女的事实上，即使在针对姓名先验进行调优校正之后，已知方向评分比请求方向评分产生更准确的标签。经过先验校正的分数，更好的方向取决于事实本身而非模板，并且在挖掘的事实上会发生反转，这些事实中的显著实体是父母而非子女。在评估中，当子女的前向事实被扣留时，使用已知方向标签训练的学生在其训练查询上的开放式准确性比使用先验校正的反向标签训练的学生提高了13到15个百分点。在生成的答案通过词汇相似性匹配到固定姓名列表后，学生几乎重现了所有选定的标签。其准确性在很大程度上遵循标签质量。标签优势在未筛选的查询以及在不插入正确答案的情况下检索候选答案时仍然成立。我们的研究结果表明，方向性验证通过提供更准确的训练目标，减轻了从教师生成的答案向学生转移错误的问题。代码可在以下网址获取：此 https URL。
英文摘要
Knowledge distillation aims to transfer the factual knowledge of large language models to smaller models for efficient deployment. Yet a teacher may recall a relation in one direction while failing to generate the answer in the reverse direction. Distillation from its generated answers can therefore propagate this directional limitation to the student. The same teacher can nevertheless recognize such an answer by scoring the relation in the direction it knows. We introduce directional label distillation, in which frozen teachers score candidate answers in that known direction and the best-scoring candidate becomes the student's training target. On facts about parents and their children, known-direction scoring yields more accurate labels than scoring the requested direction, even after tuned corrections for name priors. With prior-corrected scores, the better direction depends on the facts rather than the template, and reverses on mined facts whose notable entity is the parent rather than the child. With the evaluated children's forward facts withheld, students trained on known-direction labels improve open-ended accuracy on their trained queries by 13 to 15 points over students trained on prior-corrected reverse labels. After generated answers are matched to a fixed name list by lexical similarity, students reproduce nearly all selected labels. Their accuracy largely follows label quality. The label advantage holds on unscreened queries and when candidates are retrieved without inserting correct answers. Our findings show that directional verification mitigates the transfer of errors from teacher-generated answers to students by providing more accurate training targets. Code is available at https://github.com/js-lee-AI/directional-verification.
Comments29 pages, 7 figures, 31 tables
