CoT-Valve:长度可压缩的思维链调优
CoT-Valve: Length-Compressible Chain-of-Thought Tuning
AI总结:
针对思维链推理成本高的问题,提出CoT-Valve调优与推理策略,通过控制参数空间方向实现推理链长度弹性调节,在降低推理开销的同时保持性能,优于提示控制方法。
摘要:
思维链(Chain-of-Thought)显著增强了模型的推理能力,但也因长链导致推理成本大幅增加。基于“简单任务下推理路径易压缩、复杂任务下难压缩”的观察,我们探索用单一模型弹性控制推理路径长度的可行性,从而动态降低推理模型的推理开销。我们提出一种名为CoT-Valve的调优与推理策略,旨在让模型生成不同长度的推理链。为此,我们提出识别参数空间中的一个方向,通过操纵该方向可有效控制生成的思维链长度。此外,我们证明该特性对压缩推理链具有价值。我们为相同问题构建了从长到短的推理链数据集,并探索了CoT-Valve的两种增强策略:(1)精确的长度可压缩思维链调优方法,(2)渐进式链长度压缩方法。实验表明,CoT-Valve成功实现了链的可控性与可压缩性,且性能优于基于提示的控制方法。我们将该方法应用于QwQ-32B-Preview,在GSM8K上将推理链从741个token压缩至225个token,性能仅轻微下降(从95.07%降至94.92%);在AIME上将推理链从6827个token压缩至4629个token,仅增加一个错误答案。
英文摘要:
Chain-of-Thought significantly enhances a model's reasoning capability, but it also comes with a considerable increase in inference costs due to long chains. With the observation that the reasoning path can be easily compressed under easy tasks but struggle on hard tasks, we explore the feasibility of elastically controlling the length of reasoning paths with only one model, thereby reducing the inference overhead of reasoning models dynamically based on task difficulty. We introduce a new tuning and inference strategy named CoT-Valve, designed to allow models to generate reasoning chains of varying lengths. To achieve this, we propose to identify a direction in the parameter space that, when manipulated, can effectively control the length of generated CoT. Moreover, we show that this property is valuable for compressing the reasoning chain. We construct datasets with chains from long to short for the same questions and explore two enhanced strategies for CoT-Valve: (1) a precise length-compressible CoT tuning method, and (2) a progressive chain length compression approach. Our experiments show that CoT-Valve successfully enables controllability and compressibility of the chain and shows better performance than the prompt-based control. We applied this method to QwQ-32B-Preview, reducing reasoning chains on GSM8K from 741 to 225 tokens with a minor performance drop (95.07% to 94.92%) and on AIME from 6827 to 4629 tokens, with only one additional incorrect answer.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。