arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23711cs.CV
收藏

统一的生成与理解模型能否在不同输出模态间保持语义等价性?

Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?

Tencent Youtu Lab(腾讯云图实验室);Xiamen University(厦门大学);Computing Lab, Department of Artificial Intelligence, School of Informatics(计算实验室,人工智能系,信息学院)

Hongbo Jiang, Jie Li, Yunhang Shen, Pingyang Dai, Xing Sun, Haoyu Cao, Liujuan Cao

AI总结:

本研究探讨统一生成与理解模型在不同输出模态间保持语义等价性的能力,发现其在视觉回答任务中存在性能下降问题,原因在于跨模态语义对齐的失效。

摘要:

统一多模态大语言模型(U-MLLMs)在单一架构中集成了理解和生成能力。然而,现有评估通常分别评估这些能力,忽略了语义等价性,即无论输出模态如何,都能表现出一致的推理结果。在本工作中,我们研究当前U-MLLMs是否满足这一前提。我们发现,尽管模型在文本推理方面表现稳健,但当要求以图像模态呈现相同结果时,却无法保持语义等价性。为严格诊断这一差异,我们引入VGUBench,一个将推理逻辑与生成保真度解耦的框架。VGUBench包含三个诊断任务:(1)文本生成理解,建立文本响应推理准确性的基准;(2)视觉生成理解,评估生成正确答案的视觉响应能力;(3)视觉渲染控制任务,评估将显式视觉描述直接渲染为图像的能力,而无需复杂推理。我们的评估揭示了显著的差异:尽管在文本理解和视觉渲染方面表现强劲,U-MLLMs在要求生成视觉答案时却表现出明显的性能下降。此外,我们发现视觉回答性能与基本渲染质量之间存在微弱的相关性。这些结果表明,失败并非源于生成保真度不足,而是源于跨模态语义对齐的崩溃。我们提供了诊断见解,以在未来统一生成与理解模型中解决这一挑战。

英文摘要:

Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture. However, existing evaluations typically assess these capabilities separately, overlooking semantic equivalence, i.e., the ability to manifest consistent reasoning results regardless of the output modality. In this work, we investigate whether current U-MLLMs satisfy this premise. We observe that while models demonstrate robust textual reasoning, they fail to maintain semantic equivalence when required to render the same results in the image modality. To rigorously diagnose this discrepancy, we introduce VGUBench, a framework to decouple reasoning logic from generation fidelity. VGUBench comprises three diagnostic tasks: (1)Textual Generative Understanding, establishing a baseline for reasoning accuracy in textual response; (2)Visual Generative Understanding, evaluating the ability to generate visual responses that represent the correct answer; and (3)a Visual Rendering control task, which assesses the ability to directly render explicit visual descriptions into images without complex reasoning. Our evaluation reveals a significant disparity: despite strong performance in textual understanding and visual rendering, U-MLLMs exhibit a marked performance collapse when required to generate visual answers to questions. Furthermore, we find a negligible correlation between visual answering performance and basic rendering quality. These results suggest that the failure stems not from insufficient generation fidelity, but from a breakdown in cross-modal semantic alignment. We provide diagnostic insights to address this challenge in future Unified Generation and Understanding Models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑