UniBench：视觉推理要求重新思考视觉语言，而非仅依赖扩展
UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
AI总结：
本文提出统一评估框架UniBench，覆盖50多个VLM基准并评测59个模型，揭示规模扩展难以提升推理、关系及简单计数能力，强调数据质量与定制目标。
摘要：
研究人员已投入大量努力来扩展和改进视觉语言模型（VLM）训练方法。然而，随着基准测试数量不断增加，研究人员需要承担繁重负担：实现每一种评估协议、支付不可忽视的计算成本，并理解这些基准如何转化为有意义的进步维度。为促进对VLM进展的系统化评估，我们提出UniBench：一个统一实现的50多个VLM基准集合，涵盖从物体识别到空间感知、计数等经过仔细分类的广泛能力。我们通过评估近60个公开可用的视觉语言模型展示UniBench在衡量进展方面的效用，这些模型的训练规模最高达128亿个样本。我们发现，尽管扩展训练数据或模型规模可以提升许多视觉语言模型能力，但扩展对推理或关系理解几乎没有收益。令人惊讶的是，我们还发现当今最好的VLM在简单数字识别和计数任务（例如MNIST）上表现困难，而简单得多的网络就能解决这些任务。在规模不足的地方，我们发现更精准的干预（如数据质量或定制化学习目标）更有前景。对于实践者，我们还提供了针对给定应用选择合适VLM的指导。最后，我们发布了易于运行的UniBench代码库，包含全部50多个基准和59个模型的比较，以及一个精炼且具代表性的基准集合，可在单块GPU上于5分钟内运行。
英文摘要：
Significant research efforts have been made to scale and improve vision-language model (VLM) training approaches. Yet, with an ever-growing number of benchmarks, researchers are tasked with the heavy burden of implementing each protocol, bearing a non-trivial computational cost, and making sense of how all these benchmarks translate into meaningful axes of progress. To facilitate a systematic evaluation of VLM progress, we introduce UniBench: a unified implementation of 50+ VLM benchmarks spanning a comprehensive range of carefully categorized capabilities from object recognition to spatial awareness, counting, and much more. We showcase the utility of UniBench for measuring progress by evaluating nearly 60 publicly available vision-language models, trained on scales of up to 12.8B samples. We find that while scaling training data or model size can boost many vision-language model capabilities, scaling offers little benefit for reasoning or relations. Surprisingly, we also discover today's best VLMs struggle on simple digit recognition and counting tasks, e.g. MNIST, which much simpler networks can solve. Where scale falls short, we find that more precise interventions, such as data quality or tailored-learning objectives offer more promise. For practitioners, we also offer guidance on selecting a suitable VLM for a given application. Finally, we release an easy-to-run UniBench code-base with the full set of 50+ benchmarks and comparisons across 59 models as well as a distilled, representative set of benchmarks that runs in 5 minutes on a single GPU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
