JacobiNeRF：利用互信息梯度塑造NeRF
JacobiNeRF: NeRF Shaping with Mutual Information Gradients
AI总结：
提出JacobiNeRF，通过正则化学习动力学对齐高度相关实体的Jacobian以最大化互信息，利用二阶信息塑造NeRF表达语义协同，在极稀疏标签下高效实现2D像素与3D点间的标签传播并减轻标注负担。
摘要：
我们提出一种训练神经辐射场（NeRF）的方法，使其不仅编码场景外观，还编码场景点、区域或实体间的语义相关性，旨在捕捉它们的共同协变模式。与传统一阶光度重建目标不同，我们的方法显式地正则化学习动力学以对齐高度相关实体的Jacobian，这被证明能在随机场景扰动下最大化它们之间的互信息。通过关注这种二阶信息，当网络权重沿单个实体、区域甚至单个点的梯度改变一个delta时，我们可以塑造NeRF以表达语义上有意义的协同作用。为展示这种互信息建模的优势，我们利用由该塑造产生的场景实体协调行为，为语义分割和实例分割执行标签传播。实验表明，与无互信息塑造的NeRF相比，JacobiNeRF在2D像素和3D点间传播标注更高效，尤其在极稀疏标签情况下，从而减轻标注负担。该机制还可用于实体选择或场景修改。
英文摘要：
We propose a method that trains a neural radiance field (NeRF) to encode not only the appearance of the scene but also semantic correlations between scene points, regions, or entities -- aiming to capture their mutual co-variation patterns. In contrast to the traditional first-order photometric reconstruction objective, our method explicitly regularizes the learning dynamics to align the Jacobians of highly-correlated entities, which proves to maximize the mutual information between them under random scene perturbations. By paying attention to this second-order information, we can shape a NeRF to express semantically meaningful synergies when the network weights are changed by a delta along the gradient of a single entity, region, or even a point. To demonstrate the merit of this mutual information modeling, we leverage the coordinated behavior of scene entities that emerges from our shaping to perform label propagation for semantic and instance segmentation. Our experiments show that a JacobiNeRF is more efficient in propagating annotations among 2D pixels and 3D points compared to NeRFs without mutual information shaping, especially in extremely sparse label regimes -- thus reducing annotation burden. The same machinery can further be used for entity selection or scene modifications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
