arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.08053cs.CV
收藏

SCoFT:面向公平图像生成的自对比微调

SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation

Carnegie Mellon University(卡内基梅隆大学);Nanyang Technological University(南洋理工大学);Dongguk University(东国大学)

Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, Jean Oh

AI总结:

本文提出自对比微调方法 SCoFT 并构建跨文化理解基准 CCUB,通过利用模型自身偏见进行自改进,缓解生成图像中的文化误现与刻板印象,实验证明其优于 Stable Diffusion 基线。

摘要:

众所周知,媒体中的准确呈现能够改善内容消费者的福祉。在诸如 LAION 等大规模网络爬取数据集上训练的生成式图像模型,已知会生成带有有害刻板印象和文化误现的图像。我们通过以下方式改进生成图像中的包容性呈现:(1)与相关社群合作,收集一个具有文化代表性的数据集,我们称之为跨文化理解基准(Cross-Cultural Understanding Benchmark,CCUB);(2)提出一种新颖的自对比微调(Self-Contrastive Fine-Tuning,SCoFT)方法,该方法利用模型已知的偏见进行自我改进。SCoFT 旨在防止在小数据集上过拟合,仅编码数据中的高层信息,并使生成分布偏离预训练模型中编码的误现。我们基于参与者自选的国家文化归属,对来自 5 个不同国家的 51 名参与者进行的用户研究表明,与 Stable Diffusion 基线相比,在 CCUB 上进行微调能够持续生成具有更高文化相关性和更少刻板印象的图像,而我们的 SCoFT 技术进一步提升了这一效果。

英文摘要:

Accurate representation in media is known to improve the well-being of the people who consume it. Generative image models trained on large web-crawled datasets such as LAION are known to produce images with harmful stereotypes and misrepresentations of cultures. We improve inclusive representation in generated images by (1) engaging with communities to collect a culturally representative dataset that we call the Cross-Cultural Understanding Benchmark (CCUB) and (2) proposing a novel Self-Contrastive Fine-Tuning (SCoFT) method that leverages the model's known biases to self-improve. SCoFT is designed to prevent overfitting on small datasets, encode only high-level information from the data, and shift the generated distribution away from misrepresentations encoded in a pretrained model. Our user study conducted on 51 participants from 5 different countries based on their self-selected national cultural affiliation shows that fine-tuning on CCUB consistently generates images with higher cultural relevance and fewer stereotypes when compared to the Stable Diffusion baseline, which is further improved with our SCoFT technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑