LLaVA-OneVision-2：迈向下一代感知智能
LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
AI总结：
提出LLaVA-OV-2模型，通过编解码流令牌化、窗口注意力和3D RoPE实现统一视频理解与时空定位，在多项基准上超越Qwen3-VL-8B。
摘要：
我们介绍LLaVA-OneVision-2（LLaVA-OV-2），这是LLaVA-OneVision系列中迄今为止能力最强的视觉语言模型，在广泛的多模态基准测试中均取得了卓越性能。该模型基于原生OneVision编码器，并引入窗口注意力机制以实现高效的局部计算，同时保持原生分辨率。其关键进展是编解码流令牌化：它将压缩视频视为连续的比特成本流，其中比特成本动态决定自适应时间分组，运动残差线索选择显著空间证据到紧凑的视觉画布中。这种分配将有限的令牌预算集中在包含事件的内容上，相比固定图片组，实现了更稳定的长视频令牌压缩。共享的3D RoPE进一步将编解码画布、采样帧和图像置于统一的时空坐标系中。此外，我们围绕大规模开放监督构建了LLaVA-OV-2数据和训练栈：约800万重新标注的视频样本用于预训练，400万样本的空间语料库用于微调。我们还引入了JumpScore，这是一个针对高频、密集重复运动中的细粒度定位的时空定位基准，填补了现有视频评估的空白。LLaVA-OV-2的一项突出能力是其在视频理解、时空定位、空间定位和操作轨迹推理上的统一感知。在JumpScore上，LLaVA-OneVision-2-8B达到74.9 JumpScore mAP，比Qwen3-VL-8B（30.1）高出44.8分；在同一基准的匹配视觉令牌预算下，编解码流输入相比帧采样在时空定位上提升9.7分。在标准基准上，LLaVA-OneVision-2-8B在视频任务上平均比Qwen3-VL-8B高出4.3分，在空间任务上高出5.3分，在跟踪任务上平均J&F高出15.6分。
英文摘要：
We introduce LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-language model in the LLaVA-OneVision series to date, achieving superior performance across a broad range of multimodal benchmarks. The model builds on a native OneVision-Encoder and incorporates Windowed Attention for efficient local computation while maintaining native resolution. Its key advance is codec-stream tokenization: it treats compressed video as a continuous bit-cost stream, where bit-cost dynamics determine adaptive temporal groups, and motion-residual cues select salient spatial evidence into compact visual canvases. This allocation concentrates a limited token budget on event-bearing content, enabling more stable long-video token compression than fixed groups of pictures. A shared 3D RoPE further places codec canvases, sampled frames, and images in a unified spatiotemporal coordinate system. Furthermore, we build the LLaVA-OV-2 data and training stack around large-scale open supervision: approximately 8M re-captioned video samples for pretraining, a 4M-sample spatial corpus for fine-tuning. We also introduce JumpScore, a temporal-localization benchmark targeting fine-grained grounding in high-frequency, densely repeated motion, a regime underrepresented by existing video evaluations. A standout capability of LLaVA-OV-2 is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-trace reasoning. On JumpScore, LLaVA-OneVision-2-8B reaches 74.9 JumpScore mAP, surpassing Qwen3-VL-8B (30.1) by +44.8 points; under matched visual-token budgets on the same benchmark, codec-stream inputs improve temporal grounding over frame sampling by +9.7 points. Across standard benchmarks, LLaVA-OneVision-2-8B further outperforms Qwen3-VL-8B by +4.3 average points on video tasks, +5.3 on spatial tasks, and +15.6 average J&F on tracking tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
