LoRi: 用于隐式推理的低秩蒸馏
LoRi: Low-Rank Distillation for Implicit Reasoning
AI总结：
提出低秩蒸馏框架，通过对齐师生模型在共享低秩张量子空间中的隐状态推理轨迹，提升大型语言模型的隐式思维链推理能力。
摘要：
隐式思维链方法旨在将推理内化到大型语言模型中，但通常表现不如显式思维链提示。我们通过实验发现，隐状态推理轨迹具有低秩结构。基于此观察，我们提出了一种低秩蒸馏框架，通过使用一阶和二阶统计量，在共享的低秩张量子空间中对齐教师和学生轨迹来传递推理能力。得到的公式捕捉了推理的全局结构，同时支持紧凑的潜在推理过程。我们在多个模型家族（包括LLaMA和Qwen）上，在不同规模下对数学推理基准进行了评估。我们的方法持续提升了性能，尤其是在具有挑战性的多步任务上，接近显式思维链的准确率，并优于先前的隐式思维链蒸馏方法。
英文摘要：
Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
