$π$-StepNFT: 更宽的空间需要更细的步骤在线RL用于基于流的VLAs
$π$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
AI总结:
$π$-StepNFT通过分步负向感知微调方法,在在线强化学习中提升基于流的VLAs的性能和泛化能力。
摘要:
基于流的视觉-语言-动作(VLA)模型在具身控制中表现出色,但在多步采样过程中面临不可行的似然,阻碍了在线强化学习。我们提出了\textbf{\textit{$\boldsymbolπ$-StepNFT}}(分步负向感知微调),一种无需批评者和似然的框架,只需每个优化步骤进行一次前向传递,并消除辅助价值网络。我们发现,更宽的探索空间需要更细粒度、分步的指导以实现对齐。实证上,$π$-StepNFT在LIBERO上解锁了潜在能力,具有竞争力的少样本鲁棒性。此外,它在ManiSkill上实现了更优的泛化,优于基于价值的基线,在OOD场景中通过防止过拟合多模态特征而表现更佳。这一特性提供了一种可扩展的解决方案,有望应用于复杂的现实世界应用。
英文摘要:
Flow-based vision-language-action (VLA) models have emerged as a powerful paradigm for embodied control, but multi-step denoising makes policy-gradient online reinforcement learning (RL) difficult because the final-action likelihood induced by the sampler is not directly tractable. We identify an exploration--supervision mismatch within the denoising trajectory, where stochastic differential equation (SDE) sampling explores a wider local stochastic transition space while supervision derived from the final denoised output does not directly use the adjacent transitions observed during sampling. We propose $π$-StepNFT (Step-wise Negative-aware Fine-Tuning), a flow-native online RL framework that couples wider stochastic exploration with finer step-wise supervision without computing the final-action likelihood. $π$-StepNFT treats observed SDE transitions as supervision units, using variance-normalized transition errors to rank mirrored positive and negative velocity branches and map rollout feedback to local velocity updates. On LIBERO, starting from shared few-shot SFT initializations, $π$-StepNFT raises average success from 57.6\% to 92.0\% for $π_0$ and from 77.1\% to 94.0\% for $π_{0.5}$. On ManiSkill, it achieves the highest reported mean success among the evaluated RL baselines on five test-only visual perturbations. Our project homepage is available at https://wangst0181.github.io/pi-StepNFT/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。