异构相机的前馈式新视角合成

中文摘要 AI 辅助

前馈式新视角合成近年来在稀疏位姿图像上展现了令人鼓舞的结果，但大多数现有方法假设上下文视图和目标视图共享固定的相机家族。这一同质相机假设在多传感器实际系统中会失效，因为透视相机、鱼眼相机和全景相机可能共存，且目标投影在训练时可能未见。我们研究跨异构中心相机的前馈式新视角合成，并识别出由标记化引入的关键歧义：一个视觉标记聚合了依赖于投影的像素射线束，而现有相机编码主要暴露绝对射线或标记中心关系。为解决此问题，我们将标记中心相对相机位置编码与提出的局部射线图相结合，后者是一种标记级表示，显式描述每个标记汇总的补丁内射线分布。我们进一步提出投影感知的二维旋转位置编码，用射线诱导的角度坐标替代原始图像网格坐标，使相对位置推理在不同相机投影间对齐。这些组件共同将多样相机视为共享射线空间中的校准采样，而非分离的视觉域。在具有异构相机系统的ScanNet++上，我们的方法在混合相机评估下优于相机条件基线，并展示了对全景视图的零样本泛化能力。

英文摘要

Feed-forward novel view synthesis has recently shown promising results from sparse posed images, but most existing methods assume that context and target views share a fixed camera family. This homogeneous-camera assumption breaks in practical multi-sensor systems, where perspective, fisheye, and panoramic cameras may coexist and where the target projection may be unseen during training. We study feed-forward NVS across heterogeneous central cameras and identify a key ambiguity introduced by tokenization: a visual token aggregates a projection-dependent bundle of pixel rays, while existing camera encodings mainly expose absolute rays or token-center relations. To address this, we combine token-center relative Camera Positional Encodings and proposed local raymaps, a token-level representation that explicitly describes the intra-patch ray distribution summarized by each token. We further propose projection-aware 2D RoPE, which replaces raw image-grid coordinates with ray-induced angular coordinates so that relative positional reasoning is aligned across camera projections. Together, these components treat diverse cameras as calibrated samplings of a shared ray space rather than separate visual domains. On ScanNet++ with heterogeneous-camera system, our method improves over camera-conditioned baselines under mixed-camera evaluation and demonstrates zero-shot generalization to panoramic views.

发表机构

• Shanghai Jiaotong University（上海交通大学）
• Nanyang Technological University（南洋理工大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Accepted at NeuralIPS 2026