在 Option 诱导的抽象 MDP 中通过变分同态学习时间抽象
Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs
AI总结：
针对显式 CoT 推理成本高的问题，提出基于 VMOC、MDP 同态理论与 SFT 冷启动的潜在 option 推理框架，并在逻辑推理和运动控制任务中验证有效。
摘要：
大语言模型（LLM）已通过显式的思维链（CoT）提示展现出卓越的推理能力，但生成这些逐步的文本解释在计算上既昂贵又缓慢。为克服这一问题，我们旨在开发一个高效的隐式推理框架，使模型在潜在空间中“思考”，而无需为每一步生成显式文本。我们提出，可在分层强化学习框架内，将这些潜在思考建模为时间延展的抽象动作，即 options。为了有效学习一个多样化的 options 库作为潜在嵌入，我们首先提出 Variational Markovian Option Critic（VMOC），这是一种在 HiT-MDP 框架内使用变分推断的离策略算法。为给将这些 options 用作抽象推理空间提供严格基础，我们扩展了连续 MDP 同态理论。这证明了在简化的抽象潜在空间中学习策略（VMOC 正适合该任务）能够保持对原始复杂问题求解的最优性。最后，我们提出一种冷启动流程，利用监督微调（SFT）数据将人类推理示范蒸馏到该潜在 option 空间中，从而为模型的推理能力提供丰富初始化。大量实验表明，我们的方法在复杂逻辑推理基准和具有挑战性的运动任务上均取得强劲表现，验证了该框架是一种为语言与控制学习抽象技能的有原则方法。
英文摘要：
Large Language Models (LLMs) have shown remarkable reasoning ability through explicit Chain-of-Thought (CoT) prompting, but generating these step-by-step textual explanations is computationally expensive and slow. To overcome this, we aim to develop a framework for efficient, implicit reasoning, where the model "thinks" in a latent space without generating explicit text for every step. We propose that these latent thoughts can be modeled as temporally-extended abstract actions, or options, within a hierarchical reinforcement learning framework. To effectively learn a diverse library of options as latent embeddings, we first introduce the Variational Markovian Option Critic (VMOC), an off-policy algorithm that uses variational inference within the HiT-MDP framework. To provide a rigorous foundation for using these options as an abstract reasoning space, we extend the theory of continuous MDP homomorphisms. This proves that learning a policy in the simplified, abstract latent space, for which VMOC is suited, preserves the optimality of the solution to the original, complex problem. Finally, we propose a cold-start procedure that leverages supervised fine-tuning (SFT) data to distill human reasoning demonstrations into this latent option space, providing a rich initialization for the model's reasoning capabilities. Extensive experiments demonstrate that our approach achieves strong performance on complex logical reasoning benchmarks and challenging locomotion tasks, validating our framework as a principled method for learning abstract skills for both language and control.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
