PixelWorld：我们距离将一切感知为像素还有多远？
PixelWorld: How Far Are We from Perceiving Everything as Pixels?
AI总结：
该研究提出“将一切感知为像素（PEAP）”的统一感知范式，构建PixelWorld基准测试其性能，发现其语义理解效果与词元方法相当，推理任务性能可通过思维链提示弥补，为多模态学习提供新评估框架。
摘要：
当前，智能体语言模型愈发需要与包含紧密交织的视觉和文本信息的真实世界环境交互，且交互方式通常是通过原始相机像素，而非经过单独处理的图像和分词后的文本。这一转变凸显了统一感知范式的必要性。为探究这一构想，我们研究了“将一切感知为像素（Perceive Everything as Pixels，PEAP）”，并推出了PixelWorld基准——该基准将自然语言、表格、数学和图表输入渲染到共享的像素空间中。 跨多个基准的实验表明，PEAP在语义理解任务上的性能与基于词元的方法相当，这表明视觉Transformer无需显式分词即可部分捕获全局文本语义。相比之下，数学和代码等推理密集型任务出现了显著的性能下降，不过思维链（Chain-of-Thought）提示可通过弥补缺失的符号结构来缩小这一差距。我们进一步发现，当视觉和文本信息紧密融合时，将所有内容表示为像素可简化预处理流程，并避免跨模态错位。因此，PixelWorld为评估统一视觉-语言模型提供了一个系统且实用的框架，并推动了基于像素的多模态学习的进一步探索。
英文摘要：
Recent agentic language models increasingly need to interact with real-world environments that contain tightly intertwined visual and textual information, often through raw camera pixels rather than separately processed images and tokenized text. This shift highlights the need for a unified perception paradigm. To investigate this idea, we explore Perceive Everything as Pixels (PEAP) and introduce PixelWorld, a benchmark that renders natural-language, tabular, mathematical, and diagrammatic inputs into a shared pixel space. Experiments across multiple benchmarks show that PEAP achieves comparable performance to token-based approaches on semantic understanding tasks, suggesting that vision transformers can partially capture global textual semantics without explicit tokenization. In contrast, reasoning-intensive tasks such as mathematics and code show notable performance degradation, although Chain-of-Thought prompting helps mitigate this gap by compensating for missing symbolic structure. We further find that when visual and textual information are closely integrated, representing everything as pixels simplifies preprocessing and avoids cross-modal misalignment. PixelWorld thus provides a systematic and practical framework for evaluating unified vision--language models and facilitates further exploration of pixel-based multimodal learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
