arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22734cs.CV
收藏

多模态模型中的非对称个性化特征

Asymmetric Idiosyncrasies in Multimodal Models

University of Southern California(南加州大学);New York University(纽约大学)

Muzi Tao, Chufan Shi, Huijuan Wang, Shengbang Tong, Xuezhe Ma

AI总结:

本文研究了多模态模型中描述模型的个性化特征及其对文本到图像模型的影响,发现生成图像丢失了描述中的关键变化,提出了一种新的量化方法。

摘要:

在本文中,我们研究了描述模型中的个性化特征及其对文本到图像模型的下游影响。我们设计了一种系统分析:给定一个生成的描述或对应的图像,我们训练神经网络来预测起源于哪个描述模型。我们的结果表明,文本分类的准确率非常高(99.70%),这表明描述模型嵌入了独特的风格签名。相比之下,这些签名在生成的图像中几乎消失,即使对于最先进的Flux模型,分类准确率也降至至多50%。为了更好地理解这种跨模态差异,我们进一步分析数据,发现生成的图像未能保留描述中关键的变化,例如细节水平的差异、对颜色和纹理的强调以及场景中物体分布的分布。总体而言,我们的基于分类的框架提供了一种新的方法,用于量化描述模型的风格个性化特征以及文本到图像系统的提示遵循能力。

英文摘要:

In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70\%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50\% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑