This article introduces a benchmark designed to evaluate the capabilities of multimodal models in analyzing and interpreting images. The benchmark focuses on seven key visual aspects: main object, additional objects, background, detail, dominant colors, style, and viewpoint. A dataset of 14,580 images, generated from diverse text prompts, was used to assess the performance of seven leading multimodal models. These models were evaluated on their ability to accurately identify and describe each visual aspect, providing insights into their strengths and weaknesses for comprehensive image understanding. The findings of this benchmark have significant implications for the development and selection of multimodal models for various image analysis tasks.
The University of Hong Kong（香港大学）；JD Explore Academy（京东探索研究院）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
University of Helsinki（赫尔辛基大学）；Politecnico di Torino（都灵理工大学）；Universiteit Utrecht（乌得勒支大学）；Université Bretagne Sud（南布列塔尼大学）；University of Copenhagen（哥本哈根大学）；University Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；University of Lorraine（洛林大学）
Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study
Department of Chemistry and Bioscience（化学与生物科学系）；Aalborg University（奥尔堡大学）；Department of Architecture, Design and Media Technology（建筑、设计与媒体技术系）；Visual Analysis and Perception Lab（视觉分析与感知实验室）
Cornell University（康奈尔大学）；Caltech（加州理工学院）；Google（谷歌）；University of Edinburgh（爱丁堡大学）
Recent progress in self-supervised learning has resulted in models that are capable of extracting rich representations from image collections without requiring any explicit label supervision. However, to date the vast majority of these approaches have restricted themselves to training on standard benchmark datasets such as ImageNet. We argue that fine-grained visual categorization problems, such as plant and animal species classification, provide an informative testbed for self-supervised learning. In order to facilitate progress in this area we present two new natural world visual classificat…
Hanoi University of Science and Technology（河内理工大学）；Griffith University（格里菲斯大学）；Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Deakin University（迪肯大学）；The University of Queensland（昆士兰大学）
In recent years, visual forgery has reached a level of sophistication that humans cannot identify fraud, which poses a significant threat to information security. A wide range of malicious applications have emerged, such as fake news, defamation or blackmailing of celebrities, impersonation of politicians in political warfare, and the spreading of rumours to attract views. As a result, a rich body of visual forensic techniques has been proposed in an attempt to stop this dangerous trend. In this paper, we present a benchmark that provides in-depth insights into visual forgery and visual forens…
University of Washington（华盛顿大学）；Google Research（谷歌研究院）；Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）
We introduce the new Birds-to-Words dataset of 41k sentences describing fine-grained differences between photographs of birds. The language collected is highly detailed, while remaining understandable to the everyday observer (e.g., "heart-shaped face," "squat body"). Paragraph-length descriptions naturally adapt to varying levels of taxonomic and visual distance---drawn from a novel stratified sampling approach---with the appropriate level of detail. We propose a new model called Neural Naturalist that uses a joint image encoding and comparative module to generate comparative language, and ev…
Compressed Video Quality Enhancement: Classifying and Benchmarking over Standards
Faculty of Electronics and Telecommunications, VNU University of Engineering and Technology（电子与电信学院，越南工程大学）；Department of Computer Science, National Yang Ming Chiao Tung University（计算机科学系，国家阳明交通大学）
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
University College London（伦敦大学学院）；The University of Queensland（昆士兰大学）；Southeast University（东南大学）；University of Science and Technology of China（中国科学技术大学）；Auckland University of Technology（奥克兰理工大学）
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Microsoft Research（微软研究院）；Ohio State University（俄亥俄州立大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Northeastern University（东北大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Chinese University of Hong Kong（香港中文大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Georgia Institute of Technology（佐治亚理工学院）
提出 TemporalBench 基准，通过约 10K 高质量人工标注视频问答对评估多模态视频模型的细粒度时序理解能力，发现 GPT-4o 准确率仅 38.5%，并提出 MBA 指标纠正多选问答偏差。
CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；First Affiliated Hospital of Anhui University of Chinese Medicine（安徽中医药大学第一附属医院）
Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学海德堡医学院）；Helmholtz Imaging（亥姆霍兹成像）；Heidelberg University Hospital（海德堡大学医院）
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
School of Computer Science and Engineering, School of Intelligence Science and Engineering and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China（东南大学计算机科学与工程学院、智能科学与工程学院及新一代人工智能技术及其跨学科应用关键实验室，中国）；Wangxuan Institute of Computer Technology, Peking University, China（北京大学王轩计算机技术研究所，中国）；University of Copenhagen, Denmark（丹麦哥本哈根大学）
ShiftedBronzes: Benchmarking and Analysis of Domain Fine-Grained Classification in Open-World Settings
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；School of Archaeology, Jilin University（吉林大学考古学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MoE, China（教育部知识驱动人机智能工程研究中心）
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院，中国）；Alibaba Group（阿里巴巴集团）；School of Computer Science and Engineering, School of Intelligence Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China（东南大学计算机科学与工程学院、智能科学与工程学院以及新一代人工智能技术及其交叉应用关键实验室，中国）；Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China（北京大学王轩计算机技术研究所、多媒体信息处理国家重点实验室，中国）；University of Copenhagen, Denmark（丹麦哥本哈根大学）
RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios
Department of Electronic Engineering, BNRist, Tsinghua University（清华大学电子工程系、北京信息科学与技术国家研究中心）；Zhongguancun Academy（中关村学院）；Amap, Alibaba Group（阿里巴巴集团高德地图）
University of Alberta（阿尔伯塔大学）；Valeo Vision Systems（法雷奥视觉系统公司）；Cairo University（开罗大学）
Semantic segmentation benefits robotics related applications especially autonomous driving. Most of the research on semantic segmentation is only on increasing the accuracy of segmentation models with little attention to computationally efficient solutions. The few work conducted in this direction does not provide principled methods to evaluate the different design choices for segmentation. In this paper, we address this gap by presenting a real-time semantic segmentation benchmarking framework with a decoupled design for feature extraction and decoding methods. The framework is comprised of d…
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（高等技术学院）；Champalimaud Centre for the Unknown（尚帕利莫未知中心）
A defining characteristic of intelligent systems is the ability to make action decisions based on the anticipated outcomes. Video prediction systems have been demonstrated as a solution for predicting how the future will unfold visually, and thus, many models have been proposed that are capable of predicting future frames based on a history of observed frames~(and sometimes robot actions). However, a comprehensive method for determining the fitness of different video prediction models at guiding the selection of actions is yet to be developed. Current metrics assess video prediction models bas…
A video dataset that is designed to study fine-grained categorisation of pedestrians is introduced. Pedestrians were recorded "in-the-wild" from a moving vehicle. Annotations include bounding boxes, tracks, 14 keypoints with occlusion information and the fine-grained categories of age (5 classes), sex (2 classes), weight (3 classes) and clothing style (4 classes). There are a total of 27,454 bounding box and pose labels across 4222 tracks. This dataset is designed to train and test algorithms for fine-grained categorisation of people, it is also useful for benchmarking tracking, detection and …
East China Normal University（华东师范大学）；Xiaohongshu Inc.（小红书）；KLATASDS, MOE, China（中国教育部统计与数据科学重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes
GADE Union (Global AI Data Experts Union)（GADE联盟（全球人工智能数据专家联盟））；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；New York University（纽约大学）；Cambridge University（剑桥大学）；The University of Hong Kong（香港大学）
Stanford University（斯坦福大学）；University of California San Diego（加州大学圣迭戈分校）；Simon Fraser University（西蒙弗雷泽大学）；Intel AI Lab（英特尔人工智能实验室）
We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentatio…
Over the past decades, various super-resolution (SR) techniques have been developed to enhance the spatial resolution of digital images. Despite the great number of methodical contributions, there is still a lack of comparative validations of SR under practical conditions, as capturing real ground truth data is a challenging task. Therefore, current studies are either evaluated 1) on simulated data or 2) on real data without a pixel-wise ground truth. To facilitate comprehensive studies, this paper introduces the publicly available Super-Resolution Erlangen (SupER) database that includes real …
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
Max Planck Institute for Informatics（马克斯·普朗克研究所信息学研究所）；Saarland Informatics Campus（萨尔州信息学校园）；CISPA Helmholtz Center for Information Security（信息安全霍夫曼中心）；University of Freiburg（弗赖堡大学）
VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Beijing Key Laboratory of Artificial Intelligence for Education（北京人工智能教育重点实验室）；Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education（教育部智能技术与教育应用工程研究中心）
Arizona State University（亚利桑那州立大学）；Stanford University（斯坦福大学）；Mayo Clinic（梅奥诊所）
Transfer learning from supervised ImageNet models has been frequently used in medical image analysis. Yet, no large-scale evaluation has been conducted to benchmark the efficacy of newly-developed pre-training techniques for medical image analysis, leaving several important questions unanswered. As the first step in this direction, we conduct a systematic study on the transferability of models pre-trained on iNat2021, the most recent large-scale fine-grained dataset, and 14 top self-supervised ImageNet models on 7 diverse medical tasks in comparison with the supervised ImageNet model. Furtherm…
Univ. Rennes（雷恩大学）；INSA Rennes（雷恩国立应用科学学院）；IETR - UMR CNRS 6164（法国国家科学研究中心6164联合研究单元——雷恩电子与电信研究所）；DGA-MI（法国国防部军备总局-信息与系统局）
Image denoising has recently taken a leap forward due to machine learning. However, image denoisers, both expert-based and learning-based, are mostly tested on well-behaved generated noises (usually Gaussian) rather than on real-life noises, making performance comparisons difficult in real-world conditions. This is especially true for learning-based denoisers which performance depends on training data. Hence, choosing which method to use for a specific denoising problem is difficult. This paper proposes a comparative study of existing denoisers, as well as an extensible open tool that makes it…
Robotics Research Center, KCIS（KCIS机器人研究中心）；IIIT Hyderabad（海得拉巴国际信息技术学院）；Mila - Quebec AI Institute, Montreal（蒙特利尔魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
Given an image or a video captured from a monocular camera, amodal layout estimation is the task of predicting semantics and occupancy in bird's eye view. The term amodal implies we also reason about entities in the scene that are occluded or truncated in image space. While several recent efforts have tackled this problem, there is a lack of standardization in task specification, datasets, and evaluation protocols. We address these gaps with AutoLay, a dataset and benchmark for amodal layout estimation from monocular images. AutoLay encompasses driving imagery from two popular datasets: KITTI …
