学习最大化互信息以实现思维链蒸馏
Learning to Maximize Mutual Information for Chain-of-Thought Distillation
AI总结：
本文从信息瓶颈视角提出通过最大化推理生成与标签预测两任务表示特征的互信息来改进思维链蒸馏，实验在四个数据集上超越现有DSS方法。
摘要：
知识蒸馏，即将知识从大型复杂模型迁移到较小模型的技术，是迈向高效人工智能部署的关键一步。逐步蒸馏（Distilling Step-by-Step，DSS）作为一种利用思维链（chain-of-thought，CoT）蒸馏的新方法，通过赋予较小模型其较大对应模型卓越的推理能力而展现出潜力。在DSS中，被蒸馏的模型通过多任务学习框架同时获得生成推理依据和预测标签的能力。然而，DSS忽略了两个训练任务之间的内在关系，导致CoT知识与标签预测任务的有效整合不足。为此，我们从信息瓶颈（Information Bottleneck）的视角研究这两个任务的相互关系，并将其形式化为最大化两个任务表示特征的互信息。我们提出一种变分方法，利用基于学习的方法求解该优化问题。我们在四个数据集上的实验结果表明，我们的方法优于最先进的DSS。我们的发现为未来关于语言模型蒸馏以及涉及CoT的应用研究提供了有益的指导。代码可在\url{https://github.com/xinchen9/cot_distillation_ACL2024}获取。
英文摘要：
Knowledge distillation, the technique of transferring knowledge from large, complex models to smaller ones, marks a pivotal step towards efficient AI deployment. Distilling Step-by-Step~(DSS), a novel method utilizing chain-of-thought~(CoT) distillation, has demonstrated promise by imbuing smaller models with the superior reasoning capabilities of their larger counterparts. In DSS, the distilled model acquires the ability to generate rationales and predict labels concurrently through a multi-task learning framework. However, DSS overlooks the intrinsic relationship between the two training tasks, leading to ineffective integration of CoT knowledge with the task of label prediction. To this end, we investigate the mutual relationship of the two tasks from Information Bottleneck perspective and formulate it as maximizing the mutual information of the representation features of the two tasks. We propose a variational approach to solve this optimization problem using a learning-based method. Our experimental results across four datasets demonstrate that our method outperforms the state-of-the-art DSS. Our findings offer insightful guidance for future research on language model distillation as well as applications involving CoT. Codes are available at \url{https://github.com/xinchen9/cot_distillation_ACL2024}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
