通过图像提示包装实现高效的多模态推理
Token-Efficient Multimodal Reasoning via Image Prompt Packaging
AI总结：
本文提出图像提示包装方法，通过将结构化文本嵌入图像以减少文本令牌开销，并在多个数据集和模型上验证其效果，展示了在多模态系统设计中视觉编码的重要性。
摘要：
在大规模部署大型多模态语言模型时，令牌基于的推理成本限制了其扩展性，而视觉提示策略的成本-性能行为仍缺乏充分表征。我们引入图像提示包装（IPPg），一种将结构化文本直接嵌入图像以减少文本令牌开销的提示范式，并在五个数据集、三种前沿模型（GPT-4.1、GPT-4o、Claude 3.5 Sonnet）和两个任务家族（VQA和代码生成）上进行基准测试。我们推导出一种成本公式，分解了按令牌类型计算的节省，并展示了IPPg实现了35.8-91.0%的推理成本降低。尽管最大令牌压缩达96%，在许多情况下准确性仍具有竞争力，但结果高度依赖于模型和任务：GPT-4.1在CoSQL上同时实现准确性和成本提升，而Claude 3.5在多个VQA基准上导致成本增加。系统性的错误分析揭示了失败模式分类：空间推理、非英语输入和字符敏感操作最易受攻击，而结构化任务受益最大。125种配置的渲染消融研究揭示了准确性变化达10-30个百分点，确立了视觉编码选择在多模态系统设计中的首要变量地位。
英文摘要：
Deploying large multimodal language models at scale is constrained by token-based inference costs, yet the cost-performance behavior of visual prompting strategies remains poorly characterized. We introduce Image Prompt Packaging (IPPg), a prompting paradigm that embeds structured text directly into images to reduce text token overhead, and benchmark it across five datasets, three frontier models (GPT-4.1, GPT-4o, Claude 3.5 Sonnet), and two task families (VQA and code generation). We derive a cost formulation decomposing savings by token type and show IPPg achieves 35.8--91.0\% inference cost reductions. Despite token compression of up to 96\%, accuracy remains competitive in many settings, though outcomes are highly model- and task-dependent: GPT-4.1 achieves simultaneous accuracy and cost gains on CoSQL, while Claude 3.5 incurs cost increases on several VQA benchmarks. Systematic error analysis yields a failure-mode taxonomy: spatial reasoning, non-English inputs, and character-sensitive operations are most vulnerable, while schema-structured tasks benefit most. A 125-configuration rendering ablation reveals accuracy shifts of 10--30 percentage points, establishing visual encoding choices as a first-class variable in multimodal system design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
