MetaLogic:通过逻辑等价提示词评估文本生成图像模型的鲁棒性
MetaLogic: Robustness Evaluation of Text-to-Image Models via Logically Equivalent Prompts
AI总结:
针对文本生成图像模型对语义等价的不同提示词生成结果不一致的鲁棒性问题,提出无需真实图像的MetaLogic评估框架,通过蜕变测试对比等价提示生成的图像对,可高效识别细粒度逻辑不一致问题。
摘要:
近年来,文本生成图像(T2I)模型尤其是基于扩散的架构取得了进展,显著提升了生成图像的视觉质量。然而,这类模型仍存在一个关键局限:当输入提示词发生微小的语言变化时,难以保持语义一致性。尽管这类提示词对在逻辑上是等价的,却往往生成不对齐或语义不一致的图像,暴露出模型在推理和泛化方面的鲁棒性不足。为解决这一问题,我们提出MetaLogic——一种无需依赖真实图像即可检测T2I不对齐问题的新型评估框架。MetaLogic利用蜕变测试,从语法不同但语义完全一致的提示词生成图像对。通过直接比较这些图像对,该框架可识别出表明模型未能保留预期含义的不一致之处,有效诊断模型在逻辑理解上的鲁棒性问题。与现有将生成图像与单个提示词进行比较的评估方法不同,MetaLogic评估配对图像之间的语义等价性,提供了一种可扩展、无需真实标注的对齐失败识别方法。它会对这些对齐错误进行分类(例如实体遗漏、重复、位置不对齐),并找出可用于模型调试和优化的反例。我们在多个最先进的T2I模型上评估了MetaLogic,发现各类逻辑结构下均存在一致的鲁棒性失效问题。研究发现,即便是Flux.dev和DALLE-3这类最先进的文本生成图像模型,其不对齐率也分别达到59%和71%。我们的结果表明,MetaLogic不仅高效且可扩展,还能有效发现现有评估指标所忽略的细粒度逻辑不一致问题。
英文摘要:
Recent advances in text-to-image (T2I) models, especially diffusion-based architectures, have significantly improved the visual quality of generated images. However, these models continue to struggle with a critical limitation: maintaining semantic consistency when input prompts undergo minor linguistic variations. Despite being logically equivalent, such prompt pairs often yield misaligned or semantically inconsistent images, exposing a lack of robustness in reasoning and generalisation. To address this, we propose MetaLogic, a novel evaluation framework that detects T2I misalignment without relying on ground truth images. MetaLogic leverages metamorphic testing, generating image pairs from prompts that differ grammatically but are semantically identical. By directly comparing these image pairs, the framework identifies inconsistencies that signal failures in preserving the intended meaning, effectively diagnosing robustness issues in the model's logic understanding. Unlike existing evaluation methods that compare a generated image to a single prompt, MetaLogic evaluates semantic equivalence between paired images, offering a scalable, ground-truth-free approach to identifying alignment failures. It categorises these alignment errors (e.g., entity omission, duplication, positional misalignment) and surfaces counterexamples that can be used for model debugging and refinement. We evaluate MetaLogic across multiple state-of-the-art T2I models and reveal consistent robustness failures across a range of logical constructs. We find that even the SOTA text-to-image models like Flux.dev and DALLE-3 demonstrate a 59 percent and 71 percent misalignment rate, respectively. Our results show that MetaLogic is not only efficient and scalable, but also effective in uncovering fine-grained logical inconsistencies that are overlooked by existing evaluation metrics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。