OmniStream：在连续流中掌握感知、重建与行动
OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams
AI总结：
OmniStream通过统一的流视觉骨干网络，实现对视频流中感知、重建和行动的高效处理，展示了在多种任务上的通用性与竞争力。
摘要：
现代视觉代理需要能够实时在流环境中运行的表示，这些表示需要通用、因果和物理结构化。然而，当前的视觉基础模型仍然碎片化，专门用于图像语义感知、离线时间建模或空间几何。本文介绍OmniStream，一种统一的流视觉骨干网络，能够从多种视觉输入中有效感知、重建和行动。通过结合因果时空注意力和3D旋转位置嵌入（3D-RoPE），我们的模型通过持久的KV缓存实现高效的逐帧在线视频流处理。我们使用一种协同多任务框架，在29个数据集上预训练OmniStream，结合静态和时间表示学习、流几何重建以及视觉-语言对齐。广泛的评估显示，即使骨干网络严格冻结，OmniStream在图像和视频测试、流几何重建、复杂视频和空间推理以及机器人操作（训练中未见过）上也能够保持一致的竞争力。与追求基准特定主导地位不同，我们的工作展示了训练一个单一、多功能视觉骨干网络的可行性，该网络能够在语义、空间和时间推理上泛化，即更接近通用视觉理解的一步，为交互和具身代理提供更有意义的进展。
英文摘要：
Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
