PixelCraft：一个面向结构化图像高保真视觉推理的多智能体系统
PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images
AI总结：
针对MLLM在图表和几何图形上的感知与推理难题，提出PixelCraft多智能体系统，融合像素级定位、CV工具、动态讨论与图像记忆，显著提升结构化图像推理性能。
摘要：
结构化图像（例如图表和几何图形）对多模态大语言模型（MLLMs）而言仍具挑战性，因为感知失误可能级联导致错误结论。中间视觉线索能够引导推理；然而，现有基于线索的方法受限于低保真图像处理以及线性、僵化的推理模式，限制了其在复杂结构化图像任务上的有效性。本文提出 PixelCraft，这是一个新颖的多智能体系统，用于在结构化图像上进行高保真图像处理和灵活视觉推理。该系统由调度器、规划器、推理器、评论者以及一组视觉工具智能体组成。为实现高保真处理，我们构建了高质量语料库，并将一个 MLLM 微调为定位模型，其像素级定位结果与工具智能体中的传统计算机视觉（CV）算法相集成。在此基础上，PixelCraft 通过工具选择、智能体讨论和自我批评的动态三阶段工作流促进灵活视觉推理。此外，不同于以往只是追加历史图像的线性推理模式，PixelCraft 维护图像记忆，使规划器能够自适应地重访早期视觉步骤、探索备选推理分支，并在讨论过程中动态调整推理轨迹。在具有挑战性的图表和几何基准上进行的大量实验表明，PixelCraft 显著提升了先进 MLLMs 的视觉推理性能，为结构化图像推理树立了新标准。我们的代码将在 https://github.com/microsoft/PixelCraft 提供。
英文摘要：
Structured images (e.g., charts and geometric diagrams) remain challenging for multimodal large language models (MLLMs), as perceptual slips can cascade into erroneous conclusions. Intermediate visual cues can steer reasoning; however, existing cue-based methods are constrained with low-fidelity image processing and linear, rigid reasoning patterns, limiting their effectiveness on complex structured-image tasks. In this paper, we propose PixelCraft, a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images. The system comprises a dispatcher, a planner, a reasoner, critics, and a set of visual tool agents. To achieve high-fidelity processing, we construct a high-quality corpus and fine-tune an MLLM into a grounding model, whose pixel-level localizations are integrated with traditional computer vision (CV) algorithms in tool agents. Building on this foundation, PixelCraft facilitates flexible visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism. Moreover, unlike prior linear reasoning patterns that simply append historical images, PixelCraft maintains an image memory to allow the planner to adaptively revisit earlier visual steps, explore alternative reasoning branches, and dynamically adjust the reasoning trajectory during discussion. Extensive experiments on challenging chart and geometry benchmarks demonstrate that PixelCraft significantly improves visual reasoning performance for advanced MLLMs, setting a new standard for structured image reasoning. Our code will be available at https://github.com/microsoft/PixelCraft.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
