arXiv 2411.089232025-03-26cs.CVcs.LG

通过偏好优化对齐视觉对比学习模型

Aligning Visual Contrastive learning models via Preference Optimization

L3S Research Center（L3S研究中心）；Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Massachusetts Institute of Technology（麻省理工学院）

Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出用偏好优化（PO）方法训练对比学习模型，以对齐模型行为与人类偏好，增强对字体攻击和归纳偏差的鲁棒性，并缓解性别偏差，实验证明优于标准对比学习技术。

摘要：

对比学习模型通过在嵌入空间中对齐表示，已展现出捕捉语义相似性的卓越能力。然而，其性能可能受到训练数据质量及其固有偏差的限制。尽管偏好优化（PO）方法，如基于人类反馈的强化学习（RLHF）和直接偏好优化（DPO），已被应用于将生成模型与人类偏好对齐，但它们在对比学习中的应用尚未被探索。本文介绍了一种新颖的方法，使用不同的PO方法来训练对比学习模型，以分解复杂概念。我们的方法系统地使模型行为与期望偏好对齐，从而提升在目标任务上的性能。特别地，我们专注于增强模型对字体攻击和归纳偏差的鲁棒性，这些在如CLIP等对比视觉-语言模型中常见。我们的实验表明，使用PO训练的模型在性能上优于标准对比学习技术，同时保留其处理对抗性挑战的能力，并保持在其他下游任务上的准确性。这使得我们的方法非常适合需要公平性、鲁棒性以及与特定偏好对齐的任务。我们评估了我们的方法在应对图像字体攻击方面的效果，并探索其解耦性别概念和缓解性别偏差的能力，展示了我们方法的多功能性。

英文摘要：

Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Preference Optimization (PO) methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to align generative models with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using different PO methods to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method for tackling typographic attacks on images and explore its ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。