arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11802cs.CV
收藏

PhyBench:用于评估文本到图像模型的物理常识基准

PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models

OpenGVLab(开放智能机器研究实验室);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);University of Michigan(密歇根大学);Tongji University(同济大学);Nanjing University(南京大学);The University of Hong Kong(香港大学)

Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, Yu Qiao, Ping Luo

AI总结:

本文提出PhyBench,一个包含700个提示、覆盖4类物理场景的T2I评估数据集,用于评估模型物理常识;发现现有模型在物理推理上表现不足,并验证GPT-4o可有效评估该能力。

摘要:

文本到图像(T2I)模型在根据文本提示生成图像方面取得了实质性进展。然而,它们经常无法生成符合物理常识的图像,而这是世界模拟和日常任务应用中的一项关键能力。当前的T2I评估基准侧重于准确性、偏见和安全性等指标,忽视了对模型内部知识(尤其是物理常识)的评估。为解决这一问题,我们引入了PhyBench,这是一个全面的T2I评估数据集,包含跨4个主要类别(力学、光学、热力学和材料属性)的700个提示,涵盖31种不同的物理场景。我们评估了6个 prominent T2I模型,包括专有模型DALLE3和Gemini,并证明将物理原理纳入提示可增强模型生成物理准确图像的能力。我们的发现表明:(1)除了光学外,即使是先进模型也经常在各种物理场景中出错;(2)GPT-4o配合针对具体项目的评分指令,能有效评估模型对物理常识的理解,与人类评估高度一致;(3)当前T2I模型主要专注于文本到图像的转换,缺乏对物理常识的深入推理。我们呼吁更多关注T2I模型内的固有知识,而不仅仅将其作为图像生成工具。数据即将提供。

英文摘要:

Text-to-image (T2I) models have made substantial progress in generating images from textual prompts. However, they frequently fail to produce images consistent with physical commonsense, a vital capability for applications in world simulation and everyday tasks. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge, particularly physical commonsense. To address this issue, we introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties, encompassing 31 distinct physical scenarios. We assess 6 prominent T2I models, including proprietary models DALLE3 and Gemini, and demonstrate that incorporating physical principles into prompts enhances the models' ability to generate physically accurate images. Our findings reveal that: (1) even advanced models frequently err in various physical scenarios, except for optics; (2) GPT-4o, with item-specific scoring instructions, effectively evaluates the models' understanding of physical commonsense, closely aligning with human assessments; and (3) current T2I models are primarily focused on text-to-image translation, lacking profound reasoning regarding physical commonsense. We advocate for increased attention to the inherent knowledge within T2I models, beyond their utility as mere image generation tools. The data will be available soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑