arXiv 2509.085002025-09-10cs.AI

TCPO：面向有效具身决策的思维中心偏好优化

TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making

Tsinghua University（清华大学）；Meituan（美团）；Northeastern University（东北大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Meta AI；Wuhan University（武汉大学）

Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Qifan Wang, Xinyu Liu, Junhao Ruan, Zhongjian Qiao, Yifan Zhu, Yaxin Xu, Jingang Wang, Xiu Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对具身决策中VLM微调后响应迟缓与幻觉问题，提出TCPO方法，通过逐步偏好优化和动作一致性约束，在ALFWorld中平均成功率26.67%，较RL4VLM提升6%。

摘要：

将视觉语言模型（VLM）在特定情境动态任务中的有效泛化能力应用于具身人工智能仍然是一个重大挑战。尽管经过监督微调的模型能更好地与现实物理世界对齐，但在动态变化的环境中仍表现出响应迟缓与幻觉问题，需要进一步对齐。现有的后监督微调（post-SFT）方法依赖于强化学习和思维链（CoT）方法，受限于稀疏奖励和仅动作优化，导致样本效率低、一致性差和模型退化。为解决这些问题，本文提出思维中心偏好优化（TCPO）方法，用于有效的具身决策。具体而言，TCPO引入了一种逐步偏好优化方法，将稀疏奖励信号转化为更丰富的步骤样本对。它强调模型中间推理过程的对齐，缓解模型退化问题。此外，通过引入动作策略一致性约束（APC），进一步对模型输出施加一致性约束。在ALFWorld环境中的实验显示，平均成功率达到26.67%，相比RL4VLM提升了6%，验证了该方法在微调后缓解模型退化的有效性。这些结果凸显了将基于偏好的学习技术与CoT过程相结合，以增强具身智能体中视觉语言模型决策能力的潜力。

英文摘要：

Using effective generalization capabilities of vision language models (VLMs) in context-specific dynamic tasks for embodied artificial intelligence remains a significant challenge. Although supervised fine-tuned models can better align with the real physical world, they still exhibit sluggish responses and hallucination issues in dynamically changing environments, necessitating further alignment. Existing post-SFT methods, reliant on reinforcement learning and chain-of-thought (CoT) approaches, are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. To address these issues, this paper proposes Thought-Centric Preference Optimization (TCPO) for effective embodied decision-making. Specifically, TCPO introduces a stepwise preference-based optimization approach, transforming sparse reward signals into richer step sample pairs. It emphasizes the alignment of the model's intermediate reasoning process, mitigating the problem of model degradation. Moreover, by incorporating Action Policy Consistency Constraint (APC), it further imposes consistency constraints on the model output. Experiments in the ALFWorld environment demonstrate an average success rate of 26.67%, achieving a 6% improvement over RL4VLM and validating the effectiveness of our approach in mitigating model degradation after fine-tuning. These results highlight the potential of integrating preference-based learning techniques with CoT processes to enhance the decision-making capabilities of vision-language models in embodied agents.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。