论微调对思维链推理的影响
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
AI总结：
本研究探讨了微调对大型语言模型推理能力的影响，发现特定任务微调会降低思维链推理的忠实度，揭示了微调导致的模型内部机制转变。
摘要：
大型语言模型已成为通用智能的强大工具，展现出先进的自然语言处理能力，在各个领域都有应用。尽管其性能令人印象深刻，但最近的研究表明，通过人类反馈强化学习（RLHF）、监督微调（SFT）和量化低秩适配器（Q-LoRA）方法等微调策略，可以显著提升LLMs的特定任务性能。然而，先前的研究表明，虽然微调带来了显著的性能提升，但也导致了灾难性遗忘以及隐私和安全风险等挑战。为此，在理解微调对LLMs推理能力的影响方面，几乎没有相关研究。我们的研究调查了微调对LLMs推理能力的影响，解决了关于特定任务微调对整体推理能力的影响、微调对思维链（CoT）推理性能的影响，以及对CoT推理忠实度的影响等关键问题。通过探索这些维度，我们的研究展示了微调对LLM推理能力的影响，其中CoT推理的忠实度在四个数据集上平均有所下降，突显了微调过程导致的LLMs内部机制的潜在转变。
英文摘要：
Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities that find applications across diverse domains. Despite their impressive performance, recent studies have highlighted the potential for significant enhancements in LLMs' task-specific performance through fine-tuning strategies like Reinforcement Learning with Human Feedback (RLHF), supervised fine-tuning (SFT), and Quantized Low-Rank Adapters (Q-LoRA) method. However, previous works have shown that while fine-tuning offers significant performance gains, it also leads to challenges such as catastrophic forgetting and privacy and safety risks. To this end, there has been little to no work in \textit{understanding the impact of fine-tuning on the reasoning capabilities of LLMs}. Our research investigates the effect of fine-tuning on the reasoning abilities of LLMs, addressing critical questions regarding the impact of task-specific fine-tuning on overall reasoning capabilities, the influence of fine-tuning on Chain-of-Thought (CoT) reasoning performance, and the implications for the faithfulness of CoT reasonings. By exploring these dimensions, our study shows the impact of fine-tuning on LLM reasoning capabilities, where the faithfulness of CoT reasoning, on average across four datasets, decreases, highlighting potential shifts in internal mechanisms of the LLMs resulting from fine-tuning processes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
