The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available at https://davidmchan.github.io/clair/
Evaluating the Robustness of Open-Source Vision-Language Models to Domain Shift in Object Captioning
Centre for Robotic Autonomy in Demanding and Long-lasting Environments（机器人自主性在恶劣和持久环境中的研究中心）；Centre for Robotics and AI（机器人与人工智能中心）；University of Manchester（曼彻斯特大学）；Amentum Clean Energy Ltd（Amentum清洁能源有限公司）
ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Renmin University of China（中国人民大学）
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University（哈杰泰佩大学）；Aselsan Research（阿塞尔桑研究院）；Koç University（科克大学）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Amsterdam Public Health（阿姆斯特丹公共卫生研究院）
MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；CCNU（中国地质大学）
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Seoul National University（首尔大学）；Seoul National University Hospital（首尔大学医院）；Seoul National University College of Medicine（首尔大学医学院）；Sungkyunkwan University School of Medicine（成均馆大学医学院）
TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Microsoft Research（微软研究院）；Ohio State University（俄亥俄州立大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Northeastern University（东北大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Chinese University of Hong Kong（香港中文大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Georgia Institute of Technology（佐治亚理工学院）
提出 TemporalBench 基准，通过约 10K 高质量人工标注视频问答对评估多模态视频模型的细粒度时序理解能力，发现 GPT-4o 准确率仅 38.5%，并提出 MBA 指标纠正多选问答偏差。
M-A-P；Interdisciplinary Centre for Security, Reliability and Trust (SnT), Université du Luxembourg（卢森堡大学安全、可靠性与跨学科中心（SnT））；University of Manchester（曼彻斯特大学）；Zhejiang University（浙江大学）；The University of New South Wales（新南威尔士大学）；USTC（中国科学技术大学）；Dartmouth College（达特茅斯学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；NTU（南洋理工大学）
DeVAn: Dense Video Annotation for Video-Language Models
ByteDance, Inc.（字节跳动公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity
Hong Kong University of Science and Technology（香港科技大学）；Tencent Youtu Lab（腾讯优图实验室）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
University of Science and Technology of China（中国科学技术大学）；Southeast University（东南大学）；Alibaba Group（阿里巴巴集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
