基于微调函数向量的关系知识蒸馏
Relational Knowledge Distillation Using Fine-tuned Function Vectors
AI总结：
通过微调函数向量提升关系知识表示，增强语言模型的类比推理能力。
摘要：
表示概念之间的关系是智能系统理解世界的核心前提。最近使用因果中介分析的工作表明，一小组注意力头编码了上下文学习中的任务表示，这种表示被压缩为称为函数向量的紧凑表示。我们证明，仅用少量示例（约20个词对）微调函数向量，在基于关系的词完成任务上的表现优于使用原始向量，这些改进适用于小型和大型语言模型。此外，微调后的函数向量在关系词的解码性能上有所提高，并且与人类对语义关系的相似性判断有更强的对齐性。接下来，我们引入了复合函数向量——细调函数向量的加权组合——以提取关系知识并支持类比推理。在推理时，将此复合向量插入LLM激活中，显著提升了在来自认知科学和SAT基准的挑战性类比问题上的表现。我们的结果突显了激活修补作为编码和操控关系知识的可控机制的潜力，推动了大语言模型的可解释性和推理能力。
英文摘要：
Representing relations between concepts is a core prerequisite for intelligent systems to make sense of the world. Recent work using causal mediation analysis has shown that a small set of attention heads encodes task representation in in-context learning, captured in a compact representation known as the function vector. We show that fine-tuning function vectors with only a small set of examples (about 20 word pairs) yields better performance on relation-based word-completion tasks than using the original vectors derived from causal mediation analysis. These improvements hold for both small and large language models. Moreover, the fine-tuned function vectors yield improved decoding performance for relation words and show stronger alignment with human similarity judgments of semantic relations. Next, we introduce the composite function vector - a weighted combination of fine-tuned function vectors - to extract relational knowledge and support analogical reasoning. At inference time, inserting this composite vector into LLM activations markedly enhances performance on challenging analogy problems drawn from cognitive science and SAT benchmarks. Our results highlight the potential of activation patching as a controllable mechanism for encoding and manipulating relational knowledge, advancing both the interpretability and reasoning capabilities of large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
