三思而后行：大语言模型中的渐进式思维精炼
Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models
AI总结：
本文提出渐进式思维精炼框架PTR，通过弱强模型协同构建精炼数据和思维掩码微调，使大语言模型学会逐步改进回答，在十个任务上平均准确率从49.6%提升至53.5%，并展现出开放式任务中的自我改进能力。
摘要：
大语言模型（LLMs）的最新进展表明，渐进式精炼而非仅给出单一答案，能够产生更准确、更深思熟虑的输出。然而，现有方法往往严重依赖监督信号来评估先前的回答，这使得在更开放式的场景中难以有效评估输出质量。此外，这些方法通常针对特定任务设计，限制了其向新领域的泛化能力。为解决这些局限，我们提出渐进式思维精炼（Progressive Thought Refinement，PTR）框架，使LLMs能够逐步精炼其回答。PTR分两个阶段运行：（1）思维数据构建阶段：我们提出一种弱模型与强模型协同选择策略，构建高质量的渐进式精炼数据集，以确保从思维到答案的逻辑一致性，且答案在每一轮中逐步精炼。（2）思维掩码微调阶段：我们设计了一种训练结构，对“思维”进行掩码并调整损失权重，以鼓励LLMs精炼先前的思维，教会它们隐式地理解“如何改进”而非“什么是正确的”。实验结果表明，PTR在十个多样化任务上显著提升了LLM性能（平均从49.6%提升至53.5%），且无需针对特定任务进行微调。值得注意的是，在更开放式的任务中，LLMs在回答质量上也展现出超越单纯准确率的显著提升，这表明PTR真正教会了LLMs随时间自我改进。
英文摘要：
Recent advancements in large language models (LLMs) have demonstrated that progressive refinement, rather than providing a single answer, results in more accurate and thoughtful outputs. However, existing methods often rely heavily on supervision signals to evaluate previous responses, making it difficult to assess output quality in more open-ended scenarios effectively. Additionally, these methods are typically designed for specific tasks, which limits their generalization to new domains. To address these limitations, we propose Progressive Thought Refinement (PTR), a framework that enables LLMs to refine their responses progressively. PTR operates in two phases: (1) Thought data construction stage: We propose a weak and strong model collaborative selection strategy to build a high-quality progressive refinement dataset to ensure logical consistency from thought to answers, and the answers are gradually refined in each round. (2) Thought-Mask Fine-Tuning Phase: We design a training structure to mask the "thought" and adjust loss weights to encourage LLMs to refine prior thought, teaching them to implicitly understand "how to improve" rather than "what is correct." Experimental results show that PTR significantly enhances LLM performance across ten diverse tasks (avg. from 49.6% to 53.5%) without task-specific fine-tuning. Notably, in more open-ended tasks, LLMs also demonstrate substantial improvements in the quality of responses beyond mere accuracy, suggesting that PTR truly teaches LLMs to self-improve over time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
