PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models
OpenGVLab(开放智能机器研究实验室);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);University of Michigan(密歇根大学);Tongji University(同济大学);Nanjing University(南京大学);The University of Hong Kong(香港大学)
Some low-quality data and comments may mislead readers to understand the paper. We are working hard to correct these problems and resubmit the paper after making the necessary revisions
Text-to-image (T2I) models have made substantial progress in generating images from textual prompts. However, they frequently fail to produce images consistent with physical commonsense, a vital capability for applications in world simulation and everyday tasks. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge, particularly physical commonsense. To address this issue, we introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties, encompassing 31 distinct physical scenarios. We assess 6 prominent T2I models, including proprietary models DALLE3 and Gemini, and demonstrate that incorporating physical principles into prompts enhances the models' ability to generate physically accurate images. Our findings reveal that: (1) even advanced models frequently err in various physical scenarios, except for optics; (2) GPT-4o, with item-specific scoring instructions, effectively evaluates the models' understanding of physical commonsense, closely aligning with human assessments; and (3) current T2I models are primarily focused on text-to-image translation, lacking profound reasoning regarding physical commonsense. We advocate for increased attention to the inherent knowledge within T2I models, beyond their utility as mere image generation tools. The data will be available soon.
T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation
San Jose State University(圣何塞州立大学);Guilin University of Electronic Technology(桂林电子科技大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of California, Berkeley(加州大学伯克利分校)
MindWorldBench: Evaluating Mental-State-to-Behavior Reasoning in Image-to-Video Generation
Peking University(北京大学);University of Science and Technology Beijing(北京科技大学);City University of Hong Kong(香港城市大学);Nanyang Technological University(南洋理工大学)
CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(广东省新型安全智能技术重点实验室,网络空间安全研究院,计算机科学与技术学院,哈尔滨工业大学(深圳))
MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of California, Los Angeles(加州大学洛杉矶分校);Michigan State University(密歇根州立大学);University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Adelaide(阿德莱德大学);Salesforce AI Research(Salesforce人工智能研究);Microsoft(微软);Adobe Research(Adobe研究)
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Peking University(北京大学);Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);University of Toronto(多伦多大学);Sun Yat-sen University(中山大学)
$\left|\,\circlearrowright\,\boxed{\text{BUS}}\,\right|$: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles
Tredence Inc.(特伦德公司);Indian Institute of Technology Kharagpur(印度理工学院克拉格浦尔分校);Inria Paris-Rocquencourt(巴黎-罗克琴库特研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒姆研究实验室)