隐式庞加莱形变用于代理强化学习
Latent Poincaré Shaping for Agentic Reinforcement Learning
AI总结：
LaPha通过在庞加莱潜在空间中引入隐式形变，提升LLM代理在数学任务中的性能，实现自我引导的测试时间扩展和更高的准确率。
摘要：
我们提出LaPha，一种在庞加莱潜在空间中训练类似AlphaZero的LLM代理的方法。在LaPha中，搜索过程可以被视为以提示为根、向外扩展到庞加莱球边界的过程，其中负曲率提供了随着半径增加而指数增长的容量。利用双曲几何距离来验证规则的正确性，我们定义了节点电势，并通过电势差分配密集的进程奖励。我们进一步在相同的共享潜在空间上附加了一个轻量级价值头，使代理能够在几乎不增加额外开销的情况下实现自我引导的测试时间扩展。在MATH-500上，LaPha将Qwen2.5-Math-1.5B的性能从66.0%提升到88.2%。通过价值头引导的搜索，LaPha-1.5B在AIME'24上达到56.7%的准确率，而LaPha-7B在AIME'24上达到60.0%的准确率，在AIME'25上达到53.3%的准确率。
英文摘要：
We propose LaPha, a method for training AlphaZero-like LLM agents in a Poincaré latent space. Under LaPha, the search process can be visualized as a tree rooted at the prompt and growing outward from the origin toward the boundary of the Poincaré ball, where negative curvature provides exponentially increasing capacity with radius. Using hyperbolic geodesic distance to rule-verified correctness, we define a node potential and assign dense process rewards by potential differences. We further attach a lightweight value head on the same shared latent space, enabling self-guided test-time scaling with almost no additional overhead. On MATH-500, LaPha improves Qwen2.5-Math-1.5B from 66.0% to 88.2%. With value-head-guided search, LaPha-1.5B reaches 56.7% accuracy on AIME'24, and LaPha-7B further achieves 60.0% on AIME'24 and 53.3% on AIME'25.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
