从编辑器到密集几何估计器
From Editor to Dense Geometry Estimator
AI总结：
本文分析了编辑器和生成器在密集几何估计中的微调行为，提出FE2E框架，通过改进编辑模型的训练目标和精度处理，实现了零样本单目深度和法线估计的显著提升。
摘要：
利用预训练的文本到图像（T2I）生成模型的视觉先验在密集预测中已取得成功。然而，密集预测本质上是图像到图像的任务，表明图像编辑模型可能比T2I生成模型更适合微调。受此启发，我们系统分析了编辑器和生成器在密集几何估计中的微调行为。我们的发现表明，编辑模型具有固有的结构先验，使它们能够通过“细化”其固有特征更稳定地收敛，并最终比生成模型表现更好。基于这些发现，我们引入了FE2E框架，首次将基于扩散变压器（DiT）架构的先进编辑模型应用于密集几何预测。具体而言，为了适应此确定性任务，我们将编辑器的原始流匹配损失重新公式化为“一致速度”训练目标。我们还利用对数量化来解决编辑器原生BFloat16格式与任务高精度需求之间的精度冲突。此外，我们利用DiT的全局注意力，以零成本的方式在单次前向传递中联合估计深度和法线，使它们的监督信号相互增强。在不扩大训练数据的情况下，FE2E在多个数据集上实现了零样本单目深度和法线估计的显著提升。值得注意的是，其在ETH3D数据集上实现了超过35%的性能提升，并优于训练在100倍数据上的DepthAnything系列。项目页面可访问https://amap-ml.github.io/FE2E/。
英文摘要：
Leveraging visual priors from pre-trained text-to-image (T2I) generative models has shown success in dense prediction. However, dense prediction is inherently an image-to-image task, suggesting that image editing models, rather than T2I generative models, may be a more suitable foundation for fine-tuning. Motivated by this, we conduct a systematic analysis of the fine-tuning behaviors of both editors and generators for dense geometry estimation. Our findings show that editing models possess inherent structural priors, which enable them to converge more stably by ``refining" their innate features, and ultimately achieve higher performance than their generative counterparts. Based on these findings, we introduce \textbf{FE2E}, a framework that pioneeringly adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction. Specifically, to tailor the editor for this deterministic task, we reformulate the editor's original flow matching loss into the ``consistent velocity" training objective. And we use logarithmic quantization to resolve the precision conflict between the editor's native BFloat16 format and the high precision demand of our tasks. Additionally, we leverage the DiT's global attention for a cost-free joint estimation of depth and normals in a single forward pass, enabling their supervisory signals to mutually enhance each other. Without scaling up the training data, FE2E achieves impressive performance improvements in zero-shot monocular depth and normal estimation across multiple datasets. Notably, it achieves over 35\% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100$\times$ data. The project page can be accessed \href{https://amap-ml.github.io/FE2E/}{here}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
