arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12073cs.CV
收藏

借助跨文化理解基准(CCUB)数据集实现文本到图像合成模型中的公平表征

Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Dataset

Carnegie Mellon University(卡内基梅隆大学);Dongguk University(东国大学)

Zhixuan Liu, Youeun Shin, Beverley-Claire Okogwu, Youngsik Yun, Lia Coleman, Peter Schaldenbrand, Jihie Kim, Jean Oh

AI总结:

本文提出基于CCUB数据集的文化感知启动方法,通过微调Stable Diffusion与GPT-3注入视觉和语义上下文,提升生成图像的文化相关性、降低冒犯性并保持质量。

摘要:

已有研究表明,媒体中的准确表征能够提升受众的幸福感。相比之下,不准确的表征可能对观看者产生负面影响,并导致对其他文化的有害认知。为了在生成图像中实现包容性表征,我们提出了一种用于文本到图像合成的文化感知启动方法,该方法使用我们收集的一个规模较小但经过文化精选的数据集,即本文所称的 Cross-Cultural Understanding Benchmark(CCUB,跨文化理解基准)数据集,以对抗大型数据集中普遍存在的偏见。我们提出的方法包含两种微调技术:(1)通过在 CCUB 文本-图像对上微调预训练文本到图像合成模型 Stable Diffusion 来添加视觉上下文;(2)通过使用在我们的 CCUB 文化感知文本数据上训练的、经过微调的大型语言模型 GPT-3 进行自动化提示工程,来添加语义上下文。CCUB 数据集经过精选,并且我们的方法由与特定文化具有个人联系的人员进行评估。实验表明,同时使用文本和图像进行启动,能够有效提高生成图像的文化相关性并降低其冒犯性,同时保持图像质量。

英文摘要:

It has been shown that accurate representation in media improves the well-being of the people who consume it. By contrast, inaccurate representations can negatively affect viewers and lead to harmful perceptions of other cultures. To achieve inclusive representation in generated images, we propose a culturally-aware priming approach for text-to-image synthesis using a small but culturally curated dataset that we collected, known here as Cross-Cultural Understanding Benchmark (CCUB) Dataset, to fight the bias prevalent in giant datasets. Our proposed approach is comprised of two fine-tuning techniques: (1) Adding visual context via fine-tuning a pre-trained text-to-image synthesis model, Stable Diffusion, on the CCUB text-image pairs, and (2) Adding semantic context via automated prompt engineering using the fine-tuned large language model, GPT-3, trained on our CCUB culturally-aware text data. CCUB dataset is curated and our approach is evaluated by people who have a personal relationship with that particular culture. Our experiments indicate that priming using both text and image is effective in improving the cultural relevance and decreasing the offensiveness of generated images while maintaining quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑