超越模仿：用于主动潜在规划的强化学习
Beyond Imitation: Reinforcement Learning for Active Latent Planning
AI总结：
本文提出ATP-Latent方法，通过主动规划和强化学习优化潜在空间，提升链式推理的准确性和效率。
摘要：
针对高效且密集的链式思维（CoT）推理，潜在推理方法对大型语言模型（LLMs）进行微调，以将离散的语言标记替换为连续的潜在标记。这些方法相比传统语言CoT推理消耗更少的标记，并且有潜力在密集的潜在空间中进行规划。然而，当前的潜在标记通常是基于模仿语言标签进行监督的。考虑到一个问题可能有多个等价但多样的CoT标签，被动模仿任意一个可能导致较差的潜在标记表示和潜在推理策略，削弱潜在规划能力，并导致训练与测试之间的明显差距。在本工作中，我们强调在潜在标记表示空间上主动规划对于获得最优潜在推理策略的重要性。因此，我们提出了\underline{A}c\underline{t}ive Latent \underline{P}lanning方法（ATP-Latent），将潜在标记的监督过程建模为条件变分自编码器（VAE），以获得更平滑的潜在空间。此外，为了促进最合理的潜在推理策略，ATP-Latent采用强化学习（RL）并辅以一致性奖励，该奖励基于潜在标记解码内容之间的一致性，从而实现引导的RL过程。在LLaMA-1B上的实验表明，与先进的基线相比，ATP-Latent在四个基准测试中展示了+4.1%的准确率和-3.3%的标记消耗。代码可在https://github.com/zz1358m/ATP-Latent-master上获得。
英文摘要：
Aiming at efficient and dense chain-of-thought (CoT) reasoning, latent reasoning methods fine-tune Large Language Models (LLMs) to substitute discrete language tokens with continuous latent tokens. These methods consume fewer tokens compared to the conventional language CoT reasoning and have the potential to plan in a dense latent space. However, current latent tokens are generally supervised based on imitating language labels. Considering that there can be multiple equivalent but diverse CoT labels for a question, passively imitating an arbitrary one may lead to inferior latent token representations and latent reasoning policies, undermining the potential planning ability and resulting in clear gaps between training and testing. In this work, we emphasize the importance of active planning over the representation space of latent tokens in achieving the optimal latent reasoning policy. So, we propose the \underline{A}c\underline{t}ive Latent \underline{P}lanning method (ATP-Latent), which models the supervision process of latent tokens as a conditional variational auto-encoder (VAE) to obtain a smoother latent space. Moreover, to facilitate the most reasonable latent reasoning policy, ATP-Latent conducts reinforcement learning (RL) with an auxiliary coherence reward, which is calculated based on the consistency between VAE-decoded contents of latent tokens, enabling a guided RL process. In experiments on LLaMA-1B, ATP-Latent demonstrates +4.1\% accuracy and -3.3\% tokens on four benchmarks compared to advanced baselines. Codes are available on https://github.com/zz1358m/ATP-Latent-master.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
