K2P：无标签知识到提示蒸馏
K2P: Label-Free Knowledge to Prompt Distillation
- University of Georgia（佐治亚大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
K2P提出无标签知识到提示蒸馏方法，通过合成、细化及答案一致性选择提示，在无真实标签下提升冻结学生模型推理性能，并理论保证准确性。
AI中文摘要：
知识蒸馏可以通过可复用的提示将推理能力从更强的教师模型转移到冻结的学生模型，但避免权重更新并不能消除监督。在没有真实答案的情况下，教师解决方案未经验证，与教师的一致性可能奖励共同的错误。我们引入了知识到提示（K2P）方法，用于无标签知识蒸馏到提示。K2P从教师解决方案中合成可复用的指令，使用配对的教师和学生响应进行细化，并通过答案一致性指导搜索和选择。它保留自适应搜索可能低估的候选，并在预留问题上进行选择。部署仅使用冻结的学生模型和选定的提示。我们的理论分离了生成和选择的差距，并给出了在教师参考不完美的情况下，一致性引导的构建能提供准确性保证的条件。在推理任务和学生模型上，K2P整体上优于无标签替代方法，并与有监督的提示优化保持竞争力。消融实验和存档诊断评估了教师解决方案和细化的贡献，同时揭示了一致性引导选择的局限性。
英文摘要：
Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.
