任务我任何东西
Task Me Anything
摘要:
大型多模态语言模型(MLMs)的基准现在用于同时评估模型的一般能力,而不是评估特定的能力。因此,当开发人员想要确定哪些模型用于他们的应用程序时,他们会被大量的基准测试所淹没,并且仍然不确定哪个基准测试的结果最能反映他们的特定用例。本文介绍了Task-Me-Anything,一个基准生成引擎,它可以根据用户的需求生成一个基准。Task-Me-Anything维护了可视资产的可扩展分类,并且可以通过编程方式生成大量的任务实例。此外,它在计算预算内有效地解决了用户关于MLM性能的查询。它包含113 K图像,10 K视频,2K 3D对象资产,超过365个对象类别,655个属性和335个关系。它可以生成750 M图像/视频问答对,专注于评估MLM感知能力。Task-Me-Anything揭示了关键的见解:开源MLMs在对象和属性识别方面表现出色,但缺乏空间和时间理解;每个模型都具有独特的优势和弱点;较大的模型通常表现更好,尽管存在例外; GPT 4 o在识别旋转/移动对象和区分颜色方面表现出挑战。
英文摘要:
Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。