场景中人物多视角图像生成的评估框架

中文摘要 AI 辅助

近期，具备语义编辑能力的生成式图像编辑扩散Transformer（DiTs）表现出色，但在空间一致的相机角度变换方面仍存在不足。训练可执行自由形式、可提示相机角度变换的基础模型的主要瓶颈在于缺乏专门的训练数据。尽管存在针对通用3D环境和物体的多视角数据集，但仍缺少包含自然场景中固定位置人物、包含正面和侧面视图的配对多视角数据集。在非约束环境中捕获此类多相机数据在后勤上具有挑战性且不可扩展。本文首先尝试使用多个最先进的图像编辑模型合成创建该数据，但发现输出常出现幻觉，具体表现为人物头部相对于背景的旋转程度不一致，常产生环境不一致的问题。为解决此问题，本文提出头部场景旋转差异（HSRD）指标，用于定量评估围绕人物的相机移动。该指标通过将相机移动与局部头部姿态操作解耦来运行。大量实验表明，HSRD提供了评估场景中人物3D空间视差所需的流程，为可靠构建高质量多视角合成数据集铺平了道路。

英文摘要

Recent generative image-editing Diffusion Transformers (DiTs) demonstrate impressive semantic editing capabilities but still struggle with spatially consistent camera angle changes. A primary bottleneck in training foundation models to execute free-form, promptable camera angle changes is the lack of specialized training data. While multi-view datasets exist for generic 3D environments and objects, there remains an absence of paired, multi-view datasets featuring human subjects at fixed locations in natural scenes, including frontal and side-profile views. Capturing such multi-camera data in unconstrained environments is logistically challenging and unscalable. In this paper, we first experiment with multiple state-of-the-art image editing models to create this data synthetically, but find that the outputs are frequently prone to hallucinations involving how much the subject's head turns relative to the background, often producing inconsistent environments. To address this issue, we propose the Head Scene Rotation Difference (HSRD) metric to quantitatively evaluate camera movements around a person. The proposed metric operates by decoupling camera movement from localized head pose manipulation. As demonstrated by the extensive experimentation, HSRD provides the pipeline necessary to evaluate 3D spatial parallax for a person in a scene, paving the way to reliably construct high-quality multi-view synthetic datasets.

发表机构

• Princeton University（普林斯顿大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

8 pages, 3 figures