发表机构
The Pennsylvania State University; Amazon AWS AI（宾夕法尼亚州立大学; 亚马逊AWS人工智能）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对长时程VLA任务中终端奖励稀疏的问题，提出StructRL在线强化学习框架，通过可验证子任务分解构建结构化中间奖励，在RoboCasa365和LIBERO-Long上超越基线。
AI 中文摘要
视觉-语言-动作（VLA）模型在较短时程的操作任务上表现良好，但在需要根据单一指令执行多个相互依赖操作的长时程任务上仍然存在困难。在线强化学习（RL）可以通过环境交互来改进这些策略，然而许多现有方法仅在完整任务成功后才提供奖励。然而，这种终端监督是稀疏的，并且无法区分早期失败与取得实质性部分进展的轨迹。我们提出StructRL，一种在线强化学习框架，它从可验证的子任务完成中构建结构化的中间监督。StructRL将每个任务分解为可验证的子任务，仅在先决子任务完成后授予中间奖励，并根据完成进度对每个奖励进行缩放。在RoboCasa365和LIBERO-Long上，使用GR00T-N1.5和pi 0.5，StructRL持续优于所评估的在线强化学习基线。这些结果表明，可验证的结构化中间奖励改善了长时程VLA的后训练。代码可在以下网址获取：此https URL。
英文摘要
Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.
