CACTO-BIC: 通过偏采样和GPU加速轨迹优化实现可扩展的Actor-Critic学习
CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization
AI总结：
CACTO-BIC通过偏采样和GPU加速实现更高效的Actor-Critic学习，提升样本效率和计算速度，适用于高维系统和实时应用。
摘要：
轨迹优化（TO）和强化学习（RL）在解决最优控制问题方面具有互补的优势。TO能够高效地计算局部最优解，但难以处理非凸性；而RL在非凸性方面更具鲁棒性，但计算成本显著更高。CACTO（连续Actor-Critic与轨迹优化）通过学习一个热启动策略来引导TO求解器寻找低成本轨迹，以结合这两种优势。然而，可扩展性仍然是关键限制，因为系统复杂度的增加会显著提高TO的计算成本。本文引入CACTO-BIC以解决这些挑战。CACTO-BIC通过利用与局部最优策略相关的价值函数属性，偏采样初始状态以提高数据效率；同时，通过利用GPU加速来减少计算时间。实验结果表明，与CACTO相比，CACTO-BIC在样本效率和计算速度上有所提升。与PPO的比较表明，我们的方法能够在更短的时间内达到相似的解决方案。最后，针对AlienGO四足机器人进行的实验表明，CACTO-BIC能够扩展到高维系统，并适用于实时应用。
英文摘要：
Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
