UniView: 通过统一参考特征从单张图像增强新颖视角合成
UniView: Enhancing Novel View Synthesis From A Single Image By Unifying Reference Features
AI总结：
UniView通过统一参考特征提升单张图像新颖视角合成性能，采用检索增强系统和多模态大语言模型选择参考图像，并结合解耦三重注意力机制提高合成效果。
摘要：
从单张图像合成新颖视角的任务由于未观察区域存在多种解释而高度不明确。当前大多数方法倾向于从模糊先验中生成未观察区域并在输入视角附近进行插值，这通常导致严重的失真。为了解决这一限制，我们提出了一种新的模型，称为UniView，它可以利用相似物体的参考图像在视角合成过程中提供强先验信息。更具体地说，我们构建了一个检索和增强系统，并采用多模态大语言模型（MLLM）来帮助选择符合我们要求的参考图像。此外，引入了一个即插即用的适配器模块，带有多级隔离层，用于动态生成目标视角的参考特征。此外，为了保持原始输入图像的细节，我们设计了一种解耦的三重注意力机制，可以有效地对齐和整合多分支特征到合成过程中。广泛的实验表明，我们的UniView显著提高了新颖视角合成的性能，并在具有挑战性的数据集上优于最先进的方法。
英文摘要：
The task of synthesizing novel views from a single image is highly ill-posed due to multiple explanations for unobserved areas. Most current methods tend to generate unseen regions from ambiguity priors and interpolation near input views, which often lead to severe distortions. To address this limitation, we propose a novel model dubbed as UniView, which can leverage reference images from a similar object to provide strong prior information during view synthesis. More specifically, we construct a retrieval and augmentation system and employ a multimodal large language model (MLLM) to assist in selecting reference images that meet our requirements. Additionally, a plug-and-play adapter module with multi-level isolation layers is introduced to dynamically generate reference features for the target views. Moreover, in order to preserve the details of an original input image, we design a decoupled triple attention mechanism, which can effectively align and integrate multi-branch features into the synthesis process. Extensive experiments have demonstrated that our UniView significantly improves novel view synthesis performance and outperforms state-of-the-art methods on the challenging datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
