This paper presents novel benchmarks for evaluating vision-language models (VLMs) in zero-shot recognition, focusing on granularity and specificity. Although VLMs excel in tasks like image captioning, they face challenges in open-world settings. Our benchmarks test VLMs' consistency in understanding concepts across semantic granularity levels and their response to varying text specificity. Findings show that VLMs favor moderately fine-grained concepts and struggle with specificity, often misjudging texts that differ from their training data. Extensive evaluations reveal limitations in current VLMs, particularly in distinguishing between correct and subtly incorrect descriptions. While fine-tuning offers some improvements, it doesn't fully address these issues, highlighting the need for VLMs with enhanced generalization capabilities for real-world applications. This study provides insights into VLM limitations and suggests directions for developing more robust models.
Benchmarking large language models for materials synthesis: the case of atomic layer deposition
Applied Materials Division, Argonne National Laboratory（阿贡国家实验室应用材料部门）；Business and Information Systems, Argonne National Laboratory（阿贡国家实验室商业与信息系统部门）；Northwestern Center for Water Research, Northwestern University（西北大学水研究中心）
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
The Chinese University of Hong Kong（香港中文大学）；Independent（独立）；ELLIS；Technical University of Darmstadt（达姆施塔特技术大学）；Johns Hopkins University（约翰霍普金斯大学）；Monash University（墨尔本大学）
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University（广东机器感知与智能计算实验室，深圳MSU-BIT大学）；Department of Electrical and Computer System Engineering, Monash University（电子与计算机系统工程系，莫纳什大学）
OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking
Tianjin University（天津大学）；Shenzhen Institution of Advanced Technology（深圳先进技术研究院）；City University of Hong Kong（香港城市大学）；University of South Carolina（南卡罗来纳大学）
Indian Institute of Science（印度科学理工学院）；Indian Institute of Technology - Hyderabad（印度理工学院海得拉巴分校）
An examination of object recognition challenge leaderboards (ILSVRC, PASCAL-VOC) reveals that the top-performing classifiers typically exhibit small differences amongst themselves in terms of error rate/mAP. To better differentiate the top performers, additional criteria are required. Moreover, the (test) images, on which the performance scores are based, predominantly contain fully visible objects. Therefore, `harder' test images, mimicking the challenging conditions (e.g. occlusion) in which humans routinely recognize objects, need to be utilized for benchmarking. To address the concerns men…
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Department of Data Science & AI, Monash University（数据科学与人工智能系，莫纳什大学）；Faculty of Law, Monash University（法学院，莫纳什大学）
Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models
South China Normal University（华南师范大学）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Shenzhen Academy of Inspection and Quarantine（深圳检验检疫局）；Shenzhen University（深圳大学）
General Scales Unlock AI Evaluation with Explanatory and Predictive Power
Leverhulme Centre for the Future of Intelligence, University of Cambridge（剑桥大学勒沃休姆未来智能中心）；Microsoft Research Asia（微软亚洲研究院）；Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València（瓦伦西亚理工大学瓦伦西亚人工智能研究所）；Department of Engineering, University of Cambridge（剑桥大学工程系）；Department of Psychology, University of Cambridge（剑桥大学心理学系）；The Psychometrics Centre, University of Cambridge（剑桥大学心理测量中心）；Department of Theoretical and Applied Linguistics, University of Cambridge（剑桥大学理论与应用语言学系）；KU Leuven（鲁汶大学）；Educational Testing Service（教育考试服务中心）；Center for Automation and Robotics, Spanish National Research Council（西班牙国家研究委员会自动化与机器人中心）；William & Mary（威廉玛丽学院）；Princeton University（普林斯顿大学）；Carnegie Mellon University（卡内基梅隆大学）
