Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation
Electronics and Telecommunications Research Institute（电子和电信研究所）；Dept. of Electronics and Informatics, Vrije Universiteit Brussel（布鲁塞尔自由大学电子与信息系）；IMEC（imec公司）；Department of Computer Science, Texas State University（德克萨斯州立大学计算机科学系）；Embedded Computing Lab., Dept. of Computer Science and Statistics, Federal University of Santa Catarina（圣卡塔琳娜联邦大学计算机科学与统计系嵌入式计算实验室）；Licks Attorneys（利克斯律师事务所）；Federal University of Rio de Janeiro (UFRJ)（里约热内卢联邦大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东理工大学宁波数字孪生研究院）；Instituto Militar de Engenharia-IME, PGEE/PGED（军事工程学院-IME，PGEE/PGED）
Saeed Mahmoudpour, Mylene C. Q. Farias, Gi-Mun Um, Myllena A. Prado, Ismael Seidel, Leonardo de Sousa Marques, Leonardo Andrade, Shengyang Zhao, Carla L Pagliari
Benchmarking immersive media coding solutions, especially in the standardization context, requires reliable and reproducible subjective quality assessment (QA) procedures, along with objective quality metrics that remain accurate across different distortion types. This paper presents a standardized workflow for light field QA, developed and deployed in the context of JPEG Pleno standardization activities, which integrates benchmark generation, a hybrid subjective evaluation, and objective metric analysis into a common workflow. The benchmark is designed to encompass not only traditional coding-only artifacts but also distortions that arise in processing pipelines in which light field encoding is accompanied with view synthesis and reconstruction techniques. A hybrid subjective method is proposed enabling fine-grained assessment by combining reference-anchored quality rating with targeted pairwise refinement in perceptually ambiguous regions. The reliability of subjective scores is verified using statistical consistency analyses between observers of two cohorts. Finally, a large set of objective metrics is systematically evaluated in terms of global prediction accuracy, local agreement in ambiguous quality regions, and robustness across distortion families. The results show that several metrics achieve strong agreement for coding-only stimuli, but their performance consistently drops when view synthesis distortions are included. The analysis further highlights the importance of view-pooling strategy in the design of future light field quality metrics. The work provides a reproducible and standardization-ready framework for fine-grained light field QA, while identifying key limitations of current objective metrics under emerging coding pipelines. The subjectively annotated dataset is publicly available at https://plenodb.jpeg.org/lfqa/objectivecfp.
EduPersona: Benchmarking Subjective Ability Boundaries of Virtual Student Agents
School of Physical and Mathematical Sciences, Nanyang Technological University（南洋理工大学物理与数学科学学院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学教育人工智能实验室）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；State Key Laboratory of Robotics and Systems, Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）
We introduce ArtBench-10, the first class-balanced, high-quality, cleanly annotated, and standardized dataset for benchmarking artwork generation. It comprises 60,000 images of artwork from 10 distinctive artistic styles, with 5,000 training images and 1,000 testing images per style. ArtBench-10 has several advantages over previous artwork datasets. Firstly, it is class-balanced while most previous artwork datasets suffer from the long tail class distributions. Secondly, the images are of high quality with clean annotations. Thirdly, ArtBench-10 is created with standardized data collection, an…
The quality assessment (QA) of restored low light images is an important tool for benchmarking and improving low light restoration (LLR) algorithms. While several LLR algorithms exist, the subjective perception of the restored images has been much less studied. Challenges in capturing aligned low light and well-lit image pairs and collecting a large number of human opinion scores of quality for training, warrant the design of unsupervised (or opinion unaware) no-reference (NR) QA methods. This work studies the subjective perception of low light restored images and their unsupervised NR QA. Our…
Accelerating Data Processing and Benchmarking of AI Models for Pathology
Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院布罗德研究所）；Harvard-MIT（哈佛-麻省理工学院）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Sydney University（悉尼大学）；The University of Hong Kong（香港大学）
TCM-Tongue: A Standardized Tongue Image Dataset with Pathological Annotations for AI-Assisted TCM Diagnosis
Beijing Technology and Business University（北京工商大学）；Xiyuan Hospital of China Academy of Chinese Medical Sciences（中国中医科学院西苑医院）；Peking University People's Hospital（北京大学人民医院）
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign（生物工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign（电气与计算机工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Biomedical Engineering, The University of Texas at Austin（生物医学工程系，德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校）
DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学小组）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克研究所信息学研究所萨尔兰信息学校区）
本文提出DispBench基准测试工具，系统评估视差估计方法在对抗攻击和2D Common Corruptions等合成损坏下的鲁棒性，并通过迄今最广泛的分析揭示准确率、可靠性与泛化能力之间的关键相关性。
MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Quantitative Metrics for Benchmarking Medical Image Harmonization
Children’s National Hospital（儿童国家医院）；Children’s Hospital of Philadelphia（费城儿童医院）；School of Medicine and Health Sciences, George Washington University（乔治华盛顿大学医学与健康科学学院）
Cornell University（康奈尔大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Science and Technology of China（中国科学技术大学）；University of Sydney（悉尼大学）；Huazhong University of Science and Technology（华中科技大学）；Microsoft Research（微软研究院）；Texas A&M University（得克萨斯农工大学）
We present a comprehensive study and evaluation of existing single image dehazing algorithms, using a new large-scale benchmark consisting of both synthetic and real-world hazy images, called REalistic Single Image DEhazing (RESIDE). RESIDE highlights diverse data sources and image contents, and is divided into five subsets, each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full-reference metrics, to no-reference metrics, to subjective evaluation and the novel task-driven evaluation. Experiments…
