Feature4X：利用多功能高斯特征场将任意单目视频桥接到4D智能体AI
Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields
AI总结：
Feature4X提出通用框架，利用高斯泼溅将2D视频基础模型特征蒸馏为4D特征场，仅需单目视频输入，实现任意分割、场景编辑和自由形式VQA，推动4D智能体AI应用。
摘要：
近年来，2D和多模态模型通过在大规模数据集上进行大规模训练取得了显著成功。然而，将这些成果扩展到支持与复杂3D/4D场景的自由形式交互和高层语义操作仍然具有挑战性。这一困难源于大规模、带标注的3D/4D或多视图数据集的可用性有限，而这些数据集对于开放词汇和基于提示的分割、语言引导编辑以及视觉问答（VQA）等可泛化的视觉和语言任务至关重要。在本文中，我们介绍了Feature4X，一个通用框架，旨在将2D视觉基础模型的任何功能扩展到4D领域，仅使用单目视频输入，这类输入在用户生成内容中广泛可用。Feature4X中的“X”代表其多功能性，通过自适应的、模型条件的4D特征场蒸馏实现任意任务。我们框架的核心是一种动态优化策略，将多种模型能力统一到单一表示中。此外，据我们所知，Feature4X是首个利用高斯泼溅（Gaussian Splatting）将视频基础模型（如SAM2、InternVideo2）的特征蒸馏并提升到显式4D特征场的方法。我们的实验展示了新颖视角下的任意分割、几何和外观场景编辑，以及跨所有时间步的自由形式VQA，这些由大语言模型（LLM）在反馈循环中赋能。这些进展通过为可扩展、上下文感知和时空感知的系统提供基础，拓宽了智能体AI应用的范围，使沉浸式动态4D场景交互成为可能。
英文摘要：
Recent advancements in 2D and multimodal models have achieved remarkable success by leveraging large-scale training on extensive datasets. However, extending these achievements to enable free-form interactions and high-level semantic operations with complex 3D/4D scenes remains challenging. This difficulty stems from the limited availability of large-scale, annotated 3D/4D or multi-view datasets, which are crucial for generalizable vision and language tasks such as open-vocabulary and prompt-based segmentation, language-guided editing, and visual question answering (VQA). In this paper, we introduce Feature4X, a universal framework designed to extend any functionality from 2D vision foundation model into the 4D realm, using only monocular video input, which is widely available from user-generated content. The "X" in Feature4X represents its versatility, enabling any task through adaptable, model-conditioned 4D feature field distillation. At the core of our framework is a dynamic optimization strategy that unifies multiple model capabilities into a single representation. Additionally, to the best of our knowledge, Feature4X is the first method to distill and lift the features of video foundation models (e.g., SAM2, InternVideo2) into an explicit 4D feature field using Gaussian Splatting. Our experiments showcase novel view segment anything, geometric and appearance scene editing, and free-form VQA across all time steps, empowered by LLMs in feedback loops. These advancements broaden the scope of agentic AI applications by providing a foundation for scalable, contextually and spatiotemporally aware systems capable of immersive dynamic 4D scene interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
