OneVision-Encoder: 编码对齐的稀疏性作为多模态智能的基础原则
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
AI总结：
OneVision-Encoder通过编码对齐的稀疏性原则，实现高效的多模态视觉理解。
摘要：
假设。人工通用智能本质上是一个压缩问题。有效的压缩需要共鸣：深度学习在架构与数据的基本结构相一致时表现最佳。这些是根本原则。然而，现代视觉架构已偏离这些真理：视觉信号高度冗余，而辨别信息，即惊喜，是稀疏的。当前模型对密集像素网格进行统一处理，在静态背景上浪费大量计算，而不是专注于定义运动和意义的预测残差。我们主张，为了解决视觉理解问题，我们必须将架构与视频的信息论原则对齐，即编码器。 方法。OneVision-Encoder通过将预测视觉结构压缩为语义意义来编码视频。通过采用编码器碎片化，OV-Encoder放弃统一计算，专门专注于富含信号熵的3.1%-25%区域。为了在不规则令牌布局下统一空间和时间推理，OneVision-Encoder采用共享的3D RoPE，并通过在超过一百万个语义概念上进行大规模集群辨别目标进行训练，共同捕捉物体持续性和运动动态。 证据。结果验证了我们的核心假设：效率和准确性不是权衡；它们是正相关的。当集成到LLM中时，它在16个图像、视频和文档理解基准测试中 consistently 超过强大的视觉基础架构，如Qwen3-ViT和SigLIP2，尽管使用了大量较少的视觉令牌和预训练数据。值得注意的是，在视频理解任务中，OV-Encoder在Qwen3-ViT上实现了平均4.1%的改进。编码对齐的碎片级稀疏性是基础原则，使OV-Encoder成为下一代视觉通用智能的可扩展引擎。
英文摘要：
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs. Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics. Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
