IsoBench:在同构表示上 benchmark 多模态基础模型
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
AI总结:
本研究提出多模态同构表示基准数据集IsoBench,测试发现基础模型普遍偏好文本输入,图像输入会导致性能显著下降,并提出两种提示技术有效提升模型表现。
摘要:
当前的基础模型在仅使用文本输入或同时使用图像与文本输入时展现出令人印象深刻的能力。但它们的能力是否会因输入模态的不同而发生变化?在本研究中,我们提出了IsoBench(同构表示基准),这是一个包含数学、科学、算法和游戏四大领域问题的基准数据集。每个示例都配有多种同构表示形式的输入,例如视觉、文本和数学呈现形式。IsoBench提供细粒度反馈,用于诊断由表示形式导致的性能差距。在各类基础模型中,我们观察到针对同一问题,模型始终更偏好文本表示。最突出的是,在所有IsoBench问题上进行评估时,当提供图像而非文本输入时,Claude-3 Opus的性能下降28.7分;类似地,GPT-4 Turbo下降18.7分,Gemini Pro下降14.9分。最后,我们提出了两种提示技术:IsoCombination(同构组合)和IsoScratchPad(同构草稿板),它们通过考虑不同输入表示的组合以及不同表示之间的转换来提升模型性能。
英文摘要:
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose $\textbf{IsoBench}$, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple $\textbf{isomorphic representations}$ of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Finally, we present two prompting techniques, $\textit{IsoCombination}$ and $\textit{IsoScratchPad}$, which improve model performance by considering combinations of, and translations between, different input representations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。