基于解码器的感知知识蒸馏
Decoder-based Sense Knowledge Distillation
AI总结：
本文提出DSKD框架，通过整合词汇资源提升解码器式LLM的知识蒸馏性能，使生成模型能继承结构化语义并保持高效训练。
摘要：
大型语言模型（LLMs）学习上下文嵌入，能够捕捉丰富的语义信息，但往往忽视了结构化的词义知识，如词义和关系。先前的工作表明，将词义词典纳入可以提高编码器模型的知识蒸馏效果，但将其应用于解码器作为生成模型仍具挑战性。在本文中，我们介绍了基于解码器的感知知识蒸馏（DSKD），这是一种将词汇资源整合到解码器式LLM训练中的框架，无需在推理时进行词典查找。在多样化的基准测试中，广泛实验表明DSKD显著提高了解码器的知识蒸馏性能，使生成模型能够继承结构化的语义，同时保持高效的训练。
英文摘要：
Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
