通过动态剪枝和知识蒸馏实现高效的数学推理模型
Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillation
AI总结：
本文提出通过动态剪枝和知识蒸馏优化方法，提升大型语言模型在数学推理任务中的效率与性能。
摘要：
随着深度学习的快速发展，大型语言模型在复杂的数学方程求解等推理任务中展现出强大的能力。然而，其显著的计算和存储成本阻碍了实际部署。本文提出了一种轻量级优化方法，该方法结合了动态注意力头剪枝与知识蒸馏。该方法利用权重范数和熵的组合，动态评估多头注意力机制中每个注意力头的重要性，并实时剪枝冗余头以减少计算开销。为缓解性能下降，知识蒸馏将信息从原始模型转移到剪枝后的学生模型，使较小的模型能够保持推理能力。在Math23k和ASDiv-A上进行的实验验证了所提方法的有效性。例如，在Math23k上使用30%的剪枝率时，参数减少了18.7%，推理速度提高了27.5%，FLOPs减少了19.3%，而准确率仅下降了0.7%（从84.4%到83.7%）。这些结果表明，该方法在保持强大推理性能的同时实现了显著的效率提升，为高效部署大型语言模型在数学推理任务中提供了实用的解决方案。
英文摘要：
With the rapid development of deep learning, large language models have shown strong capabilities in complex reasoning tasks such as mathematical equation solving. However, their substantial computational and storage costs hinder practical deployment. This paper proposes a lightweight optimization method that integrates dynamic attention head pruning with knowledge distillation. The approach dynamically evaluates the importance of each attention head in the multi-head attention mechanism using a combination of weight norms and entropy, and prunes redundant heads in real time to reduce computational overhead. To mitigate performance degradation, knowledge distillation transfers information from the original model to the pruned student, enabling the smaller model to preserve reasoning ability. Experiments conducted on both Math23k and ASDiv-A verify the effectiveness of the proposed method. For example, on Math23k with a 30% pruning ratio, parameters are reduced by 18.7%, inference speed is improved by 27.5%, FLOPs are reduced by 19.3%, and accuracy drops only 0.7% (from 84.4% to 83.7%). These results demonstrate that the method achieves substantial efficiency gains while maintaining strong reasoning performance, providing a practical solution for efficient deployment of large language models in mathematical reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
