FindIt：面向通用多模态大语言模型的格式感知视觉检测基准
FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
AI总结：
提出首个全面评估通用多模态大语言模型在可提示定位能力上的基准，涵盖四种核心任务，并标准化输入输出格式，揭示模型对格式约束的敏感性。
摘要：
多模态大语言模型（MLLMs）主要在自由形式的视觉语言任务（如视觉问答、图像描述和摘要）上进行评估。然而，它们的实际应用正在迅速扩展到更结构化的计算机视觉场景，用户提示模型执行以定位为中心的任务（如目标检测），通常是在更大的智能体或决策系统中。尽管发生了这种转变，但目前还没有标准化的基准来系统地大规模评估这些能力。在这项工作中，我们引入了第一个专门设计用于评估通用MLLMs可提示定位能力的全面基准。我们的基准涵盖四个核心任务类别：目标检测、指代表达检测、实例级检测和基于视频的检测。为了实现一致和公平的评估，我们开发了一个统一框架，标准化输入，强制可解析的边界框输出，并定义了跨任务的透明评估协议。使用该套件，我们评估了多种开源和专有MLLMs，深入分析了它们的性能和局限性。除了准确性，我们还检查了模型遵守输出格式规范的能力，表明当前系统对格式约束高度敏感，并且即使面对微小变化也常常无法泛化。我们的结果突出了最先进的MLLMs在定位设置中的优势和缺点，并指出了改进多模态模型设计和评估的重要方向。
英文摘要：
Multimodal large language models (MLLMs) are predominantly evaluated on free-form vision-language tasks such as visual question answering, captioning, and summarization. However, their practical use is rapidly expanding to more structured computer vision settings, where users prompt models to perform localization-centric tasks such as object detection, often within larger agentic or decision-making systems. Despite this shift, there is currently no standardized benchmark that systematically evaluates these capabilities at scale. In this work, we introduce the first comprehensive benchmark specifically designed to assess the promptable localization abilities of generalist MLLMs. Our benchmark spans four core task categories: object detection, referring expression detection, instance-level detection, and video-based detection. To enable consistent and fair evaluation, we develop a unified framework that standardizes inputs, enforces parsable bounding box outputs, and defines transparent evaluation protocols across tasks. Using this suite, we evaluate a diverse set of open-source and proprietary MLLMs, providing an in-depth analysis of their performance and limitations. Beyond accuracy, we examine models' ability to adhere to output format specifications, showing that current systems are highly sensitive to formatting constraints and often fail to generalize even to minor variations. Our results highlight both the strengths and shortcomings of state-of-the-art MLLMs in localization settings, and point toward important directions for improving multimodal model design and evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
