arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05330cs.CV
收藏

MTPano:通过无标签集成密集预测先验实现多任务全景场景理解

MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors

Texas A\&M University(德克萨斯农工大学);Adobe(Adobe公司)

Jingdong Zhang, Xiaohang Zhan, Lingzhi Zhang, Yizhou Wang, Zhengming Yu, Jionghao Wang, Wenping Wang, Xin Li

AI总结:

本文提出MTPano,通过无标签集成密集预测先验解决全景场景理解中的数据稀缺和任务干扰问题,采用双桥网络和ERP令牌混合器提升性能。

摘要:

全面的全景场景理解对于沉浸式应用至关重要,但受限于高分辨率多任务标注的稀缺性。尽管视角基础模型通过数据扩展取得了成功,但直接将其应用于全景领域时,由于严重的几何失真和坐标系统差异往往失效。此外,球面空间中多样密集预测任务之间的内在关系尚未被深入探索。为了解决这些挑战,我们提出MTPano,一种稳健的多任务全景基础模型,通过无标签训练流程建立。首先,为克服数据稀缺,我们利用强大的视角密集先验。我们将全景图像投影到视角块中,使用现成的基础模型生成准确且无领域差距的伪标签,然后重新投影以作为块级监督。其次,为解决任务类型之间的干扰,我们将任务分为旋转不变(如深度、分割)和旋转变化(如表面法线)组。我们引入全景双桥网络,通过几何感知调节层解耦这些特征流,注入绝对位置和光线方向先验。为处理等距投影(ERP)的失真,我们加入ERP令牌混合器,随后是双分支桥网络,用于与梯度截断的交互,促进有益的跨任务信息共享,同时阻止不兼容任务属性的冲突梯度。此外,我们引入辅助任务以促进跨任务学习过程。广泛的实验表明,MTPano在多个基准上实现了最先进的性能,并在任务特定的全景专家基础模型中取得了具有竞争力的结果。

英文摘要:

Comprehensive panoramic scene understanding is critical for immersive applications, yet it remains challenging due to the scarcity of high-resolution, multi-task annotations. While perspective foundation models have achieved success through data scaling, directly adapting them to the panoramic domain often fails due to severe geometric distortions and coordinate system discrepancies. Furthermore, the underlying relations between diverse dense prediction tasks in spherical spaces are underexplored. To address these challenges, we propose MTPano, a robust multi-task panoramic foundation model established by a label-free training pipeline. First, to circumvent data scarcity, we leverage powerful perspective dense priors. We project panoramic images into perspective patches to generate accurate, domain-gap-free pseudo-labels using off-the-shelf foundation models, which are then re-projected to serve as patch-wise supervision. Second, to tackle the interference between task types, we categorize tasks into rotation-invariant (e.g., depth, segmentation) and rotation-variant (e.g., surface normals) groups. We introduce the Panoramic Dual BridgeNet, which disentangles these feature streams via geometry-aware modulation layers that inject absolute position and ray direction priors. To handle the distortion from equirectangular projections (ERP), we incorporate ERP token mixers followed by a dual-branch BridgeNet for interactions with gradient truncation, facilitating beneficial cross-task information sharing while blocking conflicting gradients from incompatible task attributes. Additionally, we introduce auxiliary tasks to fertilize the cross-task learning process. Extensive experiments demonstrate that MTPano achieves state-of-the-art performance on multiple benchmarks and delivers competitive results against task-specific panoramic specialist foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑