VTCBench: 视觉-语言模型能否通过视觉-文本压缩理解长上下文？
VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?
AI总结：
VTCBench评估视觉-文本压缩对视觉语言模型长上下文理解能力的影响，发现多数模型在处理压缩信息时表现不佳。
摘要：
视觉语言模型（VLMs）扩展上下文窗口时的计算和内存开销严重限制了其可扩展性。一个值得注意的解决方案是视觉-文本压缩（VTC），例如DeepSeek-OCR和Glyph框架，它们将长文本转换为密集的2D视觉表示，从而实现3倍至20倍的token压缩率。然而，这种高信息密度对VLMs的核心长上下文能力的影响仍未被充分研究。为解决这一差距，我们引入了首个VTC基准，并系统评估了VLMs在三个长上下文理解设置中的性能：VTC-检索，评估模型检索和汇总信息的能力；VTC-推理，要求模型推断潜在关联以在最小词汇重叠的情况下定位事实；以及VTC-记忆，测量在长期对话记忆中的综合问答能力。此外，我们建立了VTCBench-Wild以模拟多样的输入场景。我们全面评估了领先开源和专有模型在我们基准上的表现。结果表明，尽管能够解码文本信息（例如OCR）表现良好，但大多数VLMs在处理VTC信息时的长上下文理解能力却出人意料地差，无法捕捉上下文中的长关联或依赖关系。本研究对VTC有深入理解，并为设计更高效和可扩展的VLMs奠定了基础。
英文摘要：
The computational and memory overheads associated with expanding the context window of LLMs severely limit their scalability. A noteworthy solution is vision-text compression (VTC), exemplified by frameworks like DeepSeek-OCR and Glyph, which convert long texts into dense 2D visual representations, thereby achieving token compression ratios of 3x-20x. However, the impact of this high information density on the core long-context capabilities of vision-language models (VLMs) remains under-investigated. To address this gap, we introduce the first benchmark for VTC and systematically assess the performance of VLMs across three long-context understanding settings: VTC-Retrieval, which evaluates the model's ability to retrieve and aggregate information; VTC-Reasoning, which requires models to infer latent associations to locate facts with minimal lexical overlap; and VTC-Memory, which measures comprehensive question answering within long-term dialogue memory. Furthermore, we establish the VTCBench-Wild to simulate diverse input scenarios.We comprehensively evaluate leading open-source and proprietary models on our benchmarks. The results indicate that, despite being able to decode textual information (e.g., OCR) well, most VLMs exhibit a surprisingly poor long-context understanding ability with VTC-processed information, failing to capture long associations or dependencies in the context.This study provides a deep understanding of VTC and serves as a foundation for designing more efficient and scalable VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
