arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.21160cs.CV
收藏

通过几何奖励信用分配强化点-视觉-语言模型的3D理解

Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment

Northwestern Polytechnical University(西北工业大学);Southern University of Science and Technology(南方科技大学);The University of Sheffield(谢菲尔德大学);Hengqin Laboratory(横琴实验室);Tsinghua University(清华大学)

Jingkun Chen, Ruoshi Xu, Mingqi Gao, Shengda Luo, Jungong Han

AI总结:

本文提出几何奖励信用分配框架,通过分离整体监督信号并定向反馈,提升点-视觉-语言模型的3D结构对齐能力,同时引入重投影一致性项约束物理可行性,从而提升3D KPA和重投影一致性评分。

摘要:

本文提出几何奖励信用分配框架,通过分离整体监督信号并定向反馈,提升点-视觉-语言模型的3D结构对齐能力,同时引入重投影一致性项约束物理可行性,从而提升3D KPA和重投影一致性评分。

英文摘要:

Point-Vision-Language Models promise to empower embodied agents with executable spatial reasoning, yet they frequently succumb to geometric hallucination where predicted 3D structures contradict the observed 2D reality. We identify a key cause of this failure not as a representation bottleneck but as a structural misalignment in reinforcement learning, where sparse geometric tokens are drowned out by noisy and broadcasted sequence-level rewards. To resolve this causal dilution, we propose Geometric Reward Credit Assignment, a framework that disentangles holistic supervision into field-specific signals and routes them exclusively to their responsible token spans. This mechanism transforms vague feedback into precise gradient updates and effectively turns generic policy optimization into targeted structural alignment. Furthermore, we internalize physical constraints via a Reprojection-Consistency term which serves as a cross-modal verifier to penalize physically impossible geometries. Validated on a calibrated benchmark derived from ShapeNetCore, our approach bridges the reliability gap by boosting 3D KPA from 0.64 to 0.93, increasing 3D bounding box intersection over union to 0.686, and raising reprojection consistency scores to 0.852. Crucially, these gains are achieved while maintaining robust 2D localization performance, marking a meaningful step from plausible textual outputs toward physically verifiable spatial predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑