阿格斯能评判一切吗？跨领域比较视觉语言模型
Can Argus Judge Them All? Comparing VLMs Across Domains
AI总结：
研究跨领域视觉语言模型，提出ARGUS-EVAL评估框架，通过多种指标刻画模型行为，评估多个模型在下游任务表现，发现能力与可靠性导向排名有显著差异，如Qwen-2.5VL-3B-Instruct能力强，CLIP延迟和内存占用低。
摘要：
视觉语言模型（VLMs）在工业应用中越来越常用，模型选择通常由基准排名指导。这些排名很大程度上由下游任务决定，但性能相似的模型在不同数据集上行为差异大，存在能力-可靠性差距。我们提出ARGUS-EVAL评估框架，通过基准能力P(M)、跨数据集一致性CDC(M)、鲁棒性保留RR(M)和效率E(M)来刻画模型行为。我们评估了多个模型在下游任务中的表现，结果显示能力导向和可靠性导向的排名有显著差异。
英文摘要：
Vision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings. These rankings are largely determined by retrieval, captioning, and reasoning downstream tasks; however, models with similar task performance often show substantially different behavior across datasets. This creates a Capability-Reliability Gap between benchmark performance and observed model stability. We present ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs that characterizes model behavior through Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M). We evaluate CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct across retrieval, captioning, and reasoning downstream tasks. The results reveal notable differences between capability-oriented and reliability-oriented rankings. Qwen-2.5VL-3BInstruct achieves the strongest overall capability (R@1 = 82.7%, BLEU-4 = 47.2%, CIDEr = 141.6, CDC = 0.91), whereas CLIP records the lowest latency (31 ms) and memory footprint (0.9 GB).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
