Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);Qing Yuan Research Institute, Shanghai Jiao Tong University(上海交通大学清源研究院);Centre for Perceptual and Interactive Intelligence(感知与交互智能研究中心)
Recent text-to-image generative models can generate high-fidelity images from text inputs, but the quality of these generated images cannot be accurately evaluated by existing evaluation metrics. To address this issue, we introduce Human Preference Dataset v2 (HPD v2), a large-scale dataset that captures human preferences on images from a wide range of sources. HPD v2 comprises 798,090 human preference choices on 433,760 pairs of images, making it the largest dataset of its kind. The text prompts and images are deliberately collected to eliminate potential bias, which is a common issue in previous datasets. By fine-tuning CLIP on HPD v2, we obtain Human Preference Score v2 (HPS v2), a scoring model that can more accurately predict human preferences on generated images. Our experiments demonstrate that HPS v2 generalizes better than previous metrics across various image distributions and is responsive to algorithmic improvements of text-to-image generative models, making it a preferable evaluation metric for these models. We also investigate the design of the evaluation prompts for text-to-image generative models, to make the evaluation stable, fair and easy-to-use. Finally, we establish a benchmark for text-to-image generative models using HPS v2, which includes a set of recent text-to-image models from the academic, community and industry. The code and dataset is available at https://github.com/tgxs002/HPSv2 .
LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs
Institute of Image Communication and Network Engineering, Shanghai JiaoTong University(上海交通大学图像通信与网络工程研究所);University of Electronic and Science Technology of China(电子科技大学);Nanyang Technological University(南洋理工大学)
AI Tailoring: Evaluating Influence of Image Features on Fashion Product Popularity
John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院);Center for Computational Science & Engineering, Massachusetts Institute of Technology(麻省理工学院计算科学与工程中心)
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所);USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University(上海交通大学 USC-文化创意产业学院);Institut Universitaire de France (IUF)(法国大学研究院);University of Nantes(南特大学)
DEAR: Dataset for Evaluating the Aesthetics of Rendering
Color Reproduction and Synthesis Institute(颜色再现与合成研究所);Moscow Independent Research Institute of Artificial Intelligence(莫斯科独立人工智能研究 institute);Computational color photography Laboratory(计算色彩摄影实验室);University of Milano-Bicocca(米兰-比科卡大学)
Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data
School of Engineering, University of Galway(Galway大学工程学院);State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University(浙江大学极端光信息获取国家重点实验室)
GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Models
Department of Data Science, City University of Hong Kong(香港城市大学数据科学系);Department of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系)
ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学计算机技术研究院);Huawei Noah’s Ark Lab(华为诺亚实验室);School of Intelligence Science and Technology, University of Science and Technology Beijing(北京科技大学智能科学与技术学院);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)