超越像素:可视化代理的 introspective 和 interactive 地基
Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents
AI总结:
本文提出IVG框架,结合规范引导的反思与视图引导的交互,解决VLM在可视化任务中的误读与歧义问题,通过iPlotBench验证,提升问答准确度至0.81,并展示在自主探索与实时协作中的能力。
摘要:
Vision-Language Models (VLMs) 频繁误读值、幻觉细节和混淆重叠元素。当前方法仅依赖像素解释,导致 Pixel-Only Bottleneck:代理将交互图表视为静态图像,失去编码精确值的结构化规范。我们引入 Introspective and Interactive Visual Grounding (IVG),结合 (1) spec-grounded introspection,查询底层规范获取确定性证据,与 (2) view-grounded interaction,操纵视图以解决视觉歧义。为避免 VLM 偏见,我们提出 iPlotBench,包含 500 个交互 Plotly 图表和 6,706 个二元问题及真实规范。实验显示 introspection 提高数据重建保真度,结合 interaction 取得最高 QA 准确度 (0.81),在重叠几何上提升 6.7%。我们进一步在部署代理中展示 IVG,证明其在自主探索数据和与人类用户实时协作中的能力。
英文摘要:
Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。