arXiv 2601.000872025-12-31cs.ROcs.LG

具有时间约束的强化学习用于机器人运动规划

Reinforcement learning with timed constraints for robotics motion planning

Department of Mechanical Engineering, Iowa Technology Institute, University of Iowa（机械工程系，Iowa技术研究所，爱荷华大学）；Talus Renewables, Inc.（Talus Renewables公司）；Department of Industrial, Electronic, and Mechanical Engineering, Roma Tre University（工业、电子和机械工程系，罗马三大学）

Zhaoan Wang, Junchao Li, Mahdi Mohammad, Shaoping Xiao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种基于自动机的强化学习框架，用于在满足MITL时间约束的MDP和POMDP中合成策略，通过时间限制确定性广义巴克斯自动机实现时间正确性和性能优化。

摘要：

在动态和不确定的环境中运行的机器人系统越来越多地需要能够满足复杂任务序列并遵守严格时间约束的规划器。度量区间时间逻辑（MITL）提供了一个正式且表达能力强的框架，用于指定此类时间受限的要求；然而，将MITL与强化学习（RL）结合仍然具有挑战性，因为存在随机动态和部分可观测性。本文提出了一种统一的自动机构建RL框架，用于在马尔可夫决策过程（MDP）和部分可观测马尔可夫决策过程（POMDP）中合成策略，以满足MITL规范。MITL公式被转换为时间限制确定性广义巴克斯自动机（Timed-LDGBA），并与其底层决策过程同步，以构建适合Q学习的产品时间模型。一种简单但表达能力强的奖励结构强制执行时间正确性，同时允许额外的性能目标。该方法在三个模拟研究中得到验证：一个5×5网格世界作为MDP，一个10×10网格世界作为POMDP，以及一个类似办公室的服务机器人场景。结果表明，所提出的框架在随机转换下能够持续学习满足严格时间受限要求的策略，能够扩展到更大的状态空间，并在部分可观测环境中保持有效性，突显了其在时间敏感和不确定设置中可靠机器人规划的潜力。

英文摘要：

Robotic systems operating in dynamic and uncertain environments increasingly require planners that satisfy complex task sequences while adhering to strict temporal constraints. Metric Interval Temporal Logic (MITL) offers a formal and expressive framework for specifying such time-bounded requirements; however, integrating MITL with reinforcement learning (RL) remains challenging due to stochastic dynamics and partial observability. This paper presents a unified automata-based RL framework for synthesizing policies in both Markov Decision Processes (MDPs) and Partially Observable Markov Decision Processes (POMDPs) under MITL specifications. MITL formulas are translated into Timed Limit-Deterministic Generalized Büchi Automata (Timed-LDGBA) and synchronized with the underlying decision process to construct product timed models suitable for Q-learning. A simple yet expressive reward structure enforces temporal correctness while allowing additional performance objectives. The approach is validated in three simulation studies: a $5 \times 5$ grid-world formulated as an MDP, a $10 \times 10$ grid-world formulated as a POMDP, and an office-like service-robot scenario. Results demonstrate that the proposed framework consistently learns policies that satisfy strict time-bounded requirements under stochastic transitions, scales to larger state spaces, and remains effective in partially observable environments, highlighting its potential for reliable robotic planning in time-critical and uncertain settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。